Swets protests that the VA misevaluated the protester’s Lexi-Comp and F&C databases, as presented in the product demonstration. In particular, Swets contends that it demonstrated that both its quoted products reached the RFQ-required information in fewer screen selections than the VA’s count. Swets also argues that the agency’s evaluation of the number of screen selections required to reach the required information was unreasonable, disparate from how the agency treated Cox, and undocumented. Protest, July 11, 2014, at 30-33.

As explained below, we have been unable to determine from this record how the agency’s evaluators reached the number of screen selections for each of Swets’ databases at the product demonstration. In this regard, the record contains conflicting evidence, statements, and exhibits concerning how the VA conducted its evaluation. As the agency failed to maintain an evaluation record adequate to permit meaningful review, and has failed to rebut persuasive evidence presented by the protester about the number of screen selections required to demonstrate the effectiveness of Swets’ alternative databases, we sustain the protest. Although we recognize that this procurement was conducted as a request for quotations among BOA holders under FAR subpart 16.7, it is a fundamental principle of government accountability that an agency be able to produce a sufficient record to allow for a meaningful review where its procurement actions are challenged. See Resource Dimensions, LLC, B-404536, Feb. 24, 2011, 2011 CPD ¶ 50 at 6; e-LYNXX Corp., B-292761, Dec. 3, 2003, 2003 CPD ¶ 219 at 8; Checchi & Co. Consulting, Inc., B-285777, Oct. 10, 2000, 2001 CPD ¶ 132 at 6. An agency which fails to adequately document the rationale for its source selection, bears the risk that its determinations will be considered unsupported, and that absent such support, our Office may lack a basis to find that the agency had a reasonable basis for its determinations. Tiger Enters., Inc., B‑293951, July 26, 2004, 2004 CPD ¶ 141 at 2. In reviewing an agency’s procurement actions, we do not limit our review to contemporaneous evidence but consider, as appropriate, hearing testimony and the parties’ arguments and explanations. See Southwestern Marine, Inc.; Am. Sys. Eng’g Corp., B-265865.3, B-265865.4, Jan. 23, 1996, 96-1 CPD ¶ 56 at 10.

As detailed above, the solicitation established that the VA would use a product demonstration as the primary means for determining the technical merit of each vendor’s offering(s), and that the agency’s evaluation would be based on, among other things, the number of screen selections required to reach the desired information. RFQ at 22-24. When conducting the product demonstration, the agency evaluators and vendor representatives were not in the same physical location; rather, the demonstrations were conducted using screen-sharing software (e.g., “GoToMeeting,” or “WebJoin”). AR, Tab 13, Declaration of VA Demonstration Facilitator, Oct. 1, 2014, at 1. A separate audio line was also established so that the agency evaluators and vendors could communicate during the demonstration. Id. For each of the required product demonstration items, the agency evaluators instructed the vendor representatives to start from the “home screen” and navigate to the screen containing the required information. Id.

A Swets representative demonstrated the ability of the Lexi-Comp and F&C products to search for and reach the information required for each of the demonstration items. Protest, July 11, 2014, Declaration of Swets Government Contracts Director, July 11, 2014, at 3. The Swets representative states that, after navigating to the required information, he orally confirmed to the VA evaluators his count of the number of screen selections required (he also stated that the VA evaluators did not dispute his count). Id.

In contrast to Swets’ claim, the agency evaluators state that there was no verbal counting of screen selections by the Swets product representative. AR, Tab 12, Declaration of VA Evaluator J.N., July 28, 2014. The VA and Swets agree, however, that the agency evaluators did not state their count of the number of screen selections that it took in each instance to reach the required information. Protest, July 11, 2014, Declaration of Swets Government Contracts Director, July 11, 2014, at 3.

The VA evaluators kept worksheets of their product demonstration observations, including the total count of the number of screen selections for each of the 15 items that were researched during the demonstration. In most instances the agency evaluators agreed with each other about the number of screen selections needed to reach the desired information for each of the required items. AR, Tab 9, Evaluator Worksheets, May 16, 2014, at 1-30. However, for the Lexi-Comp database product, item “c” (“[s]how the mechanism of action for montelukast and provide references”), two of the evaluators counted six screens, while the third evaluator counted four to six screens. Id. at 11, 14, 17.

Swets asserts that the agency’s evaluation of Swets’ screen selections was inaccurate and unreasonably high. The protester also asserts that the agency’s evaluation of screen selections was unequal. In support of its allegations, the protester provided a declaration from the representative who conducted the product demonstration. The vendor representative states “I have used the screen/click counting methodology [the VA] claims to have used to calculate the number of screen selections/clicks required for both Lexi-Comp and F&C to access the information required by the evaluation factors, and the results of my calculations . . . establish that the screen counts reported by the [agency] evaluators were inaccurate and were higher than warranted.” Protest, Aug. 21, 2014, Declaration of Swets Government Contracts Director, Aug. 21, 2014, at 1-2.

To document his attempt at replicating the agency’s evaluation methodology, Swets provided “screen-by-screen,” “click-by-click,” walkthroughs (sometimes called “screen shots”) for each of the different products at issue. For example, with regard to demonstration item “a” (“Compare in table format the contraindications/adverse reactions of warfarin and dabigatran”), Swets showed that four screen selections were necessary for Lexi-Comp to reach the required information, in comparison to the VA’s count of seven screen selections. Id., exh. 3, Lexi-Comp Walkthrough for Item “a”. Likewise, with regard to demonstration item “j” (“Display in table format the contraindications/adverse reactions of prasugrel and ticagrelor”), Swets showed that Lexi-Comp and F&C each required four screen selections in comparison to the VA’s count of eight and six screen selections, respectively. Id., exh. 6, Lexi-Comp Walkthrough for Item “j”; exh. 7, F&C Walkthrough for Item “j”. Swets also provided screen-by-screen walkthroughs for demonstration item “c” for Lexi-Comp (three screen selections as compared to the VA’s count of six screen selections), and for demonstration item “l” for F&C (three screen selections as compared to the VA’s count of six screen selections). Id., exh. 11, Lexi-Comp Walkthrough for Item “c”; exh. 12, F&C Walkthrough for Item “l”. Swets declares that for all demonstration items, its total Lexi-Comp screen selection count is 52 as compared to the VA total screen selection count of 74, and that its F&C screen selection count is 51 as compared to the VA’s count of 61. Id., exh. 1, Screen Counts for Lexi-Comp and F&C (Swets Count vs. VA Count).

In response, the VA asserts that its evaluation was reasonable and consistent with the solicitation, and that its counting standards were consistently applied to all product demonstrations. The agency argues that the evaluators agreed to the methodology for counting screen selections prior to conducting the product demonstrations. In this regard, the agency states, in its response to the protest, that the evaluators counted the number of screen selections as each screen was being clicked on and counted transitioning to the next screen, which included all drop‑down menus and sub-menus. Contracting Officer’s Statement, Aug. 11, 2014, at 4; AR, Tab 12, Declaration of VA Evaluator J.N., July 28, 2014, at 1. Thus, the agency asserts that the evaluators accurately counted vendors’ screen selections based on this established methodology.

On this record, we cannot find that the agency’s documentation in regard to its screen selection counts is sufficient to allow us to review the reasonableness of the agency’s evaluation. The contemporaneous evaluation record consists of the evaluator worksheets which provide only the total screen selection counts. The documents subsequently submitted by the agency provide some additional information regarding the agency’s methodology, but provide no contemporaneous documentation of how the agency reached the screen selection count that it did in each instance.

In addition, the agency’s statements during the course of the protest have not been consistent about the methodology used by the evaluators. In sum, the agency essentially argues that it treated all vendors equally as to its screen selection counting, but it has no record to support the validity of its count, other than the worksheets showing the total count. In contrast, Swets has provided detailed, screen-by-screen walkthroughs of the Lexi-Comp’s and F&C’s products in support of its assertion that the agency’s screen selection counts are inaccurate. In light of the contrary information provided by the protester, the agency evaluator worksheets that document only evaluation results are not sufficient to demonstrate the reasonableness of the agency’s evaluation. In sum, we cannot find the agency’s evaluation results to be reasonable because the agency has not adequately documented the record on which its screen count results are based.  (Swets Information Services, B-410078: Oct 20, 2014)  (pdf)

Both protesters point out, and the agency concedes, that handwritten evaluation sheets from the evaluation team members, although compiled into a summary document, were not themselves retained. VA E-Mail to GAO, Jan. 22, 2014. M‑Pak asserts that “VA’s failure to create or retain this testing documentation makes it impossible for GAO or any other impartial reviewer to assess the merits of the VA’s evaluation.” M-Pak Comments at 4; see Custom Pak Comments at 2.

We disagree. Where an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for its source selection decision. Navistar Def., LLC; BAE Sys., Tactical Vehicle Sys. LP, B‑401865 et al., Dec. 14, 2009, 2009 CPD ¶ 258 at 13. The destruction of individual evaluator documents, however, does not render an agency’s evaluation unreasonable per se; rather, we will consider the record adequate if the consensus documents and source selection decision sufficiently document the agency’s rationale for the evaluations. Joint Mgmt. and Tech. Servs., B-294229, B-294229.2, Sept. 22, 2004, 2004 CPD ¶ 208 at 3-4.

Here, the evaluation record adequately detailed the strengths and weaknesses that formed the basis for the agency’s evaluation ratings for each vendor and for the source selection decision. See National Beef Packing Co., B-296534, Sept. 1, 2005, 2005 CPD ¶ 168 at 12 n.6. The agency has provided statements from the evaluators that indicate that, although the evaluators’ notes were not retained, they were taken into consideration and were compiled into the summary source selection decision. In this regard, the record indicates that:

[the program manager] was on the IPT [Integrated Product Team] and participated in the USPS testing. She consolidated the evaluation sheets for insertion into the SSD. A summary document was provided to the team during and at the conclusion of the evaluation process. At no time did any of the team state their ratings were not included.

Supp. AR, attach. 5 (Program Manager’s Statement I); see also attach. 8 (Program Manager’s Statement II).

Further, the source selection decision sets forth a detailed summary of the testing results for both the USPS and the VA testing. SSDD at 3-10. With regard to the USPS testing of M-Pak’s samples, the source selection decision indicated the following results:

FAIL
Tears apart easily due to weak adhesive
Lots of damage to bags from sorting belts, especially the Large and X‑Large bags
Does not withstand rough transit

SSDD at 4. With regard to the VA testing of M-Pak’s samples, the overall results were as follows:

FAIL – The Small and Medium do not contain an anti-static strip. The Large and X-Large bags have weak glue and are not tamper-resistant. The X-Large bag has a lighter inner liner that allows you to read the contents in the package.

SSDD at 4; see also SSDD at 12. Similarly, with regard to the VA testing of Custom Pak’s samples, the overall results were as follows:

FAIL. Bags are not tamper-resistant, contain poorly constructed side seams and mailing labels do not stick. The material of the bags is “crinkly” and may impact barcode readability through the mail stream.

SSDD at 7, 13. Further, these summary conclusions in the source selection document are followed by detailed, comprehensive evaluator findings with regard to each of the four sizes of sample bags submitted by M-Pak, Custom Pak, Star Poly, and the other vendors under each of the five evaluation categories. See, e.g., SSDD at 4-5 (M-Pak), 7-8 (Custom Pak), 9-10 (Star Poly).

In these circumstances, given the comprehensive nature of the agency’s summary evaluation documents, there is an adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for finding the protesters’ bags noncompliant with the specification requirements. This protest ground is denied.  (Custom Pak, Inc.; M-Pak, Inc., B-409308, B-409308.2, B-409308.3, B-409308.4: Mar 4, 2014)  (pdf)

Clark/F-P also challenges the State Department’s evaluation of Harbert’s final revised proposal, where the agency increased Harbert’s proposal ratings from good to excellent under Areas 1, 3, 4, and 6. The protester complains that there is no documentation in the record explaining how Harbert’s revisions adequately addressed the weaknesses the TEP had identified in Harbert’s initial proposal and why Harbert’s revised proposal merited an excellent rating under these areas. Supp. Protest at 4.

For example, in Area 3, cost management and value engineering, the TEP initially assessed Harbert’s proposal as having 2 exceptional strengths and 1 weakness, and assessed Clark/F-P’s proposal as having 5 exceptional strengths and 3 strengths. See AR, Tab 8, TEP Initial Consensus Evaluation Report, at 15, 35-36. Based on these assessments, both Harbert’s and Clark/F-P’s proposals were rated as good under Area 3. Id. After receiving revised proposals, the TEP increased Harbert’s rating to excellent, without explanation, other than the statement that the “TEP accepted that the Offeror corrected the weakness and the ranking was changed to Excellent.” See AR, Tab 9, TEP Final Consensus Evaluation Report, at 3. In this regard, Clark/F-P notes that the TEP did not identify any further strengths in Harbert’s proposal as a result of the firm’s proposal revisions.

The State Department responds that Harbert provided comprehensive explanations in its revised proposal that corresponded to each of the identified weaknesses, and that the TEP reasonably increased Harbert’s proposal ratings to reflect that there were no weaknesses in its revised proposal. Supp. AR at 11, 16.

As the State Department correctly notes, the evaluation of proposals and assignment of adjectival ratings should generally not be based upon a simple count of strengths and weaknesses, but on a qualitative assessment of the proposals consistent with the evaluation scheme. Supp. AR at 10 (citing Command Mgmt. Servs., Inc., B-310261, B-310261.2, Dec. 14, 2007, 2008 CPD ¶ 29 at 4). Adjectival ratings are merely a guide for intelligent decisionmaking. One Largo Metro LLC; Metroview Dev. Holdings, LLC; King Farm Assocs., LLC, B-404896 et al., June 20, 2011, 2011 CPD ¶ 128 at 14. However, evaluators and selection officials should reasonably consider the underlying bases for ratings, including the advantages and disadvantages associated with the specific content of competing proposals, in a manner that is fair and equitable and consistent with the terms of the solicitation. See MD Helicopters, Inc.; AgustaWestland, Inc., B-298502 et al., Oct. 23, 2006, 2006 CPD ¶ 164 at 15. Indeed, the Federal Acquisition Regulation (FAR) requires that agencies sufficiently document their judgments, including documenting the relative strengths, deficiencies, significant weakness, and risks supporting their proposal evaluations. See FAR §§ 4.801(b), 15.305(a), 15.308; Century Envtl. Hygiene, Inc., B-279378, June 5, 1998, 98-1 CPD ¶ 164 at 4.

Here, the record does not provide the underlying bases for the TEP’s decision to increase the ratings of Harbert’s proposal to excellent after revised proposals. As noted above, the TEP final consensus evaluation report merely states that the “TEP accepted that the offeror corrected the weakness and the ranking was changed to Excellent” without providing additional explanation. AR, Tab 9, TEP Final Consensus Evaluation Report, at 2-3. The SSA also testified that he had seen no written explanation for why the TEP increased Harbert’s proposal rating from good to excellent in these areas, nor did he engage in discussion on this matter. Tr. at 80, 81. Given the lack of documentation and support for the increased ratings in Harbert’s proposal, we find no basis to conclude that the TEP reasonably raised Harbert’s rating based on the firm’s revised proposal.  (Clark/Foulger-Pratt JV, B-406627, B-406627.2, Jul 23, 2012)  (pdf)

TEG contends that DOS’s past performance evaluation of BlueLaw, BSOM, Crucible, FedSys, and GCJS was unreasonable. Specifically, TEG contends that the agency did not determine whether the submitted past performance references of these offerors were relevant and permitted these offerors to provide less than the required number of references.

As a general matter, the evaluation of an offeror’s past performance, including the agency’s determination of the relevance and scope of an offeror’s performance history to be considered, is a matter within the discretion of the contracting agency, and we will not substitute our judgment for reasonably based past performance ratings. MFM Lamey Group, LLC, B-402377, Mar. 25, 2010, 2010 CPD ¶ 81 at 10; Yang Enters., Inc., Santa Barbara Applied Research, Inc, B-294605.4 et al., April 1, 2005, 2005 CPD ¶ 65 at 5. However, we will question an agency’s evaluation conclusions where they are unreasonable or undocumented. Clean Harbors Envtl. Servs., Inc., B-296176.2, Dec. 9, 2005, 2005 CPD ¶ 222 at 3. The critical question is whether the evaluation was conducted fairly, reasonably, and in accordance with the solicitation’s evaluation scheme, and whether it was based on relevant information sufficient to make a reasonable determination of the offeror’s past performance. DRS C3 Sys., LLC, B-310825, B-310825.2, Feb. 26, 2008, 2008 CPD ¶ 103 at 22.

TEG argues that the agency failed to consider the relevance of the prior contracts of the awardees in accordance with the RFP and the Federal Acquisition Regulation (FAR). An agency is required to consider, determine and document the similarity and relevance of an offeror’s past performance information as part of its past performance evaluation. See FAR § 15.305(a)(2); Clean Harbors Envtl. Servs., Inc., supra.

While DOS in its report on the protest states that it determined that the past performance of the awardees was relevant and gave examples of why this was the case, there is no contemporaneous documentation evidencing that the TEP meaningfully considered whether these offerors’ references represented “relevant corporate experience providing criminal justice related support services and associated support systems required under Section C.” See RFP § M.9.3.1.a at 268. Instead, the contemporaneous documentation reflects that the TEP merely presumed that each offerors’ identified contracts were relevant. For example, as to the relevance of BlueLaw’s past performance, the evaluation documentation only states:

BlueLaw identified their relevant past performance and was rated Exceptional by two, Excellent by six, and Good by one of their Past Performance Surveys submitted by Contracting Officers that have worked directly with them.

AR, Tab 10b, Post Discussion Consensus Report, at 15. The past performance evaluations for the other offerors included essentially identical language regarding the presumed relevance of their past performance, and showed that the TEP past performance evaluation focused primarily on the quality of the offeror’s past performance records.

As set forth above, the RFP here required the agency to consider whether an offeror and its major subcontractors demonstrated relevant corporate experience providing criminal justice related support services and associated support systems required under the work statement in section C. A review of BlueLaw’s past performance is illustrative of the problems in DOS’s evaluation. BlueLaw identified five prior contracts to demonstrate relevant experience as a prime contractor: (1) “INL Program and Outreach Support;” (2) “Enabling Human Rights For All;” (3) “Towards the Full Inclusion of People with Disabilities: Examining the Accessibility of Overseas Facilities and Programs Funded by the United States;” (4) “Paraguay Threshold Country Plan--Formalization of Economic Activities;” and (5) “Iraq Civil Civilian Advisory Support –Justice & Law Enforcement Assistance.” See AR, Tab 4, vol. 4, Past Performance Client References, at 1. There is no comtemporaneous documentation in the record that reasonably explains why this past performance meets the evaluation standards required by the RFP. In addition, as the protester has pointed out, it is not readily apparent that the identified efforts meet this requirement.

Based on the foregoing, we cannot conclude on this record that the agency’s past performance evaluation of the proposals was reasonable. Accordingly, we sustain the protest on this basis.  (The Emergence Group, B-404844.5,B-404844.6, Sep 26, 2011)  (pdf)

ITT challenges the agency’s cost and technical evaluations and maintains that, because the agency’s evaluation of proposals was not reasonable, its best value source selection also is unreasonable. We have carefully considered all of ITT’s assertions and sustain its protest for the reasons discussed below.

ITT principally maintains that, although the agency made substantial adjustments to the offerors’ proposals in performing its most probable cost evaluation, the record contains no explanation for why the adjustments were made. The protester also challenges the agency’s technical evaluation, maintaining that the Army either misevaluated its proposal or treated the offerors disparately during the technical evaluation.

As a general matter, in reviewing protests against allegedly improper evaluations, our Office examines the record to determine whether the agency’s conclusions are reasonable and consistent with the terms of the solicitation and applicable statutes and regulations. SOS Int’l, Ltd., B-402558.3, B-402558.9, June 3, 2010, 2010 CPD ¶ 131 at 2. Additionally, when an agency evaluates proposals for the award of a cost-reimbursement contract, an offeror’s proposed estimated cost of contract performance is not considered controlling since, regardless of the costs proposed by the offeror, the government is bound to pay the contractor its actual and allowable costs. Magellan Health Servs., B-298912, Jan. 5, 2007, 2007 CPD ¶ 81 at 13; Metro Machine Corp., B-295744, B-295744.2, Apr. 21, 2005, 2005 CPD ¶ 112 at 9; Federal Acquisition Regulation (FAR) § 16.301. Consequently, a cost realism analysis must be performed by the agency to determine the extent to which an offeror’s proposed costs represent what the contract costs are likely to be under the offeror’s unique technical approach, assuming reasonable economy and efficiency. FAR §§ 15.305(a)(1), 15.404-1(d)(1), (2); The Futures Group Int’l, B-281274.2, Mar. 3, 1999, 2000 CPD ¶ 147 at 3.

Finally, in considering an agency’s evaluation of cost and technical proposals in a cost reimbursement setting, it is axiomatic that the agency’s evaluation must be adequately documented, such that the record reflects the agency’s reconciliation of its technical and cost evaluation conclusions. Serco, Inc., B-298266, Aug. 9, 2006, 2006 CPD ¶ 120 at 7. Based on our review of the record, we find the agency’s evaluation unreasonable.

The record shows that, in performing its technical evaluation, the agency identified [deleted] significant strengths, [deleted] weaknesses, and no significant weaknesses or deficiencies in the ITT proposal. AR, exh. 5, at 22-25. For Lockheed, the agency’s technical evaluation identified [deleted] significant strengths, [deleted] strengths, [deleted] weaknesses, and [deleted] significant weaknesses or deficiencies. Id. at 25-28. Among the overall total of [deleted] weaknesses identified in both proposals combined, [deleted] were under the staffing approach subfactor.

In performing its most probable cost evaluation, the record shows that, for both proposals, the agency’s cost adjustments were confined to changes in the offerors’ proposed staffing approach, and that the agency took no exception to any other cost element proposed by either offeror. AR, exh. 6, at 14-19, 41-45.

Both firms proposed [deleted] quantities of staffing to perform the requirement, with ITT proposing [deleted] staff hours and Lockheed proposing [deleted] staff hours. Protester’s Comments, Nov. 17, 2011, at exh. A, attach. 3, at 11. The record further shows that the agency made significant changes to the offerors’ proposed staffing when performing its most probable cost evaluation.

With respect to ITT, the agency increased its proposed staffing in various areas by a total of [deleted] staff hours and reduced its staffing in other areas by [deleted] staff hours, for a combined change in ITT’s proposed staffing of [deleted] staff hours. Protester’s Comments, Nov. 17, 2011, at exh. A, attach. 1. These changes were made in approximately [deleted] percent of the labor categories ([deleted]) considered by the agency in its evaluation of ITT’s proposal.

In evaluating the Lockheed proposal, the agency made upward adjustments to its proposed staffing of [deleted] staff hours and downward adjustments of [deleted] staff hours, for a combined change in Lockheed’s proposed staffing of [deleted]. Protester’s Comments, Nov. 17, 2011, at exh. A, attach. 2. These changes were made in approximately [deleted] percent of the labor categories ([deleted]) considered by the agency in its evaluation of the Lockheed proposal. Id.

Despite these significant changes in the offerors’ proposed staffing for cost evaluation purposes, the record contains no explanation of why the changes were made. In particular, the changes are memorialized in Excel spreadsheets that reflect the agency’s mathematical calculations of the changes, but these spreadsheets include no narrative explanation of the reasons for the changes. AR, exhs. 16a-16e. Additionally, although the record includes a separate narrative cost report, the report makes no mention of the reasons underlying the agency’s staffing adjustments for cost evaluation purposes, and instead simply describes the agency’s approach to making the changes. For example, in the case of the Lockheed proposal, the cost report simply states, without elaboration, that:

[deleted]

AR, exh. 6, at 16. The record includes [deleted] language describing the changes made to the ITT proposal. Id. at 43.

In addition to the lack of any explanation for the extensive staffing changes in the agency’s cost evaluation materials, an examination of the agency’s technical evaluation report reveals no apparent connection between the agency’s technical evaluation findings on the one hand, and its cost evaluation findings on the other.

For example, the agency specifically identified [deleted] weaknesses in its technical evaluation of the ITT proposal relating to the firm’s use of [deleted] positions, finding that these positions (which appear to total [deleted] full time equivalents (FTEs)) are not required by the statement of work (SOW). AR, exh. 5, at 23. Notwithstanding this finding, the record shows that, in its cost evaluation, the agency did not reduce ITT’s staffing for [deleted] personnel. Protester’s Comments, Nov. 17, 2011, at exh. A, attach. 1, at 1. Thus, the agency’s cost evaluation appears inconsistent with the technical evaluation finding that these positions are not necessary to perform the SOW.

In contrast, there were areas where, in its cost evaluation, the agency concluded that Lockheed’s staffing was significantly overstated (for example, the record shows that the agency adjusted Lockheed’s proposed [deleted] personnel downward by approximately [deleted] FTEs, Protester’s Comments, Nov. 17, 2011, at exh. A, attach. 2, at 2), but the agency did not assign the proposal a weakness in its technical evaluation for this identified overstaffing, as it did in evaluating the ITT proposal in the area of computer operators. This essentially amounts to disparate treatment of the two firms, since the agency assigned a technical evaluation weakness for overstaffing to one firm’s proposal, but not to the other firm’s proposal.

In other instances, the agency made significant upward adjustments to the offerors’ proposed staffing in numerous areas, but the technical evaluation is silent with respect to the apparent weaknesses that these staffing shortages would suggest. For example, the agency increased Lockheed’s proposed staffing for [deleted] by [deleted] FTEs and its [deleted] personnel by [deleted] FTEs, Protester’s Comments, Nov. 17, 2011, at exh. A, attach. 2, at 2. Despite these significant upward adjustments to Lockheed’s staffing for cost evaluation purposes, there is no assignment of a weakness to the Lockheed proposal (or even a mention in the agency’s technical evaluation report) based on these significant staffing shortfalls.

In sum, the record shows that the agency made significant changes to the offerors’ proposed staffing in the overwhelming majority of their proposed labor categories when conducting its most probable cost evaluation, but the underlying basis for these changes is not memorialized in the record. In addition, the record shows that the agency failed to reconcile its evaluation in a manner that demonstrates consistency between its cost and technical evaluation findings. We conclude that the agency’s evaluation of proposals is unsupported in the record and, thus, is unreasonable. Correspondingly, the agency’s source selection decision, based on this fundamentally flawed evaluation, is likewise unreasonable. We therefore sustain ITT’s protest.  (ITT Systems Corporation, B-405865,B-405865.2, Jan 6, 2011)  (pdf)

Past Performance Confidence

For the past performance confidence factor, DISA concluded that Solers' proposal merited a satisfactory confidence rating, and BAH merited a substantial confidence rating. The agency states that for this factor, the agency considered the reference interviews, PPIRS data, and the personal knowledge of the evaluators.

With regard to the evaluation of the past performance references identified in the offerors' proposals, DISA concedes that it cannot produce copies of the questionnaires used during the interviews of the references, nor does it have any record of the interviews conducted with the references. SAR at 28; AR, Tab L, Cover Memorandum, at 1; Email from DISA to GAO and Protester (Jan. 27, 2011). Instead, the agency contends that the evaluation summary in the SRD reflects the information provided by the references to the evaluators. SAR at 27-28.

The SRD, however, merely lists conclusions by the evaluators concerning each offeror's past performance. See AR, Tab L, SRD, at 13-14. The record does not meaningfully document information provided by the past performance references, such as the questions asked and answers received, or what strengths or weaknesses were associated with particular past performance references. In the absence of this information, we are unable to determine whether the evaluation conclusions made by the evaluators are supported by the information provided by the references.

With regard to the PPIRS records, the record is unclear as to how DISA used this data. For Solers, the agency reviewed the PPIRS record for the incumbent CDWSG task order, and records for two contracts other than those identified by Solers in its proposal. Id.; DISA Response to GAO Questions at 9. For BAH, the agency reviewed PPIRS records for three contracts other than those identified by BAH in its proposal. AR, Tab L, SRD, at 13-14.

In its response to the protest, DISA states that the PPIRS data were used "to validate the information that was provided in the interviews and to also assist in the assignment of the overall relevance and confidence ratings." DISA Response to GAO Questions at 9. In a contemporaneous addendum to the SRD, the record includes a number of statements prepared in connection with the initial award evaluation and signed by the PPT members, which state, "I conclude that our past performance evaluation about [Solers or BAH] matches with the [PPIRS] assessment." AR, Tab L, SRD Addendum, attach. 1, PPT Members' Statements.

To the extent that the agency states that the PPIRS data was used for the purpose of the substantive evaluation of the offerors' past performance, the record provides no support for this proposition. The SRD does not discuss how the PPIRS information was used for Solers; instead, the substantive conclusions regarding the protester's performance relate solely to the undocumented interviews. See AR, Tab L, SRD, at 13. For BAH, the SRD states that "[t]he questionnaire and phone interview has identified that BAH has excellent past performance," and that "[t]he PPIRS data validate these findings." Id. at 14. Moreover, the record does not document which PPIRS records or what aspects of the contract performance records were considered by the evaluators or found to support and/or validate the ratings.

With regard to the evaluators' personal knowledge, DISA contends that although the agency received information concerning only one of BAH's references, the PPT utilized the personal knowledge of its members concerning the other two references. SAR at 30; DISA Response to GAO Questions at 8-9. The agency concedes, however, that the contemporaneous record does not discuss or otherwise document the personal knowledge used by evaluators in assessing the offerors' past performance. DISA Response to GAO Questions at 9.

Past Performance Relevancy

With regard to the past performance relevancy ratings, DISA states that its evaluation of the offerors considered the reference interviews, the personal knowledge of the evaluators, and the offerors' technical proposals. The agency found that Solers' past performance was "relevant" based on the following assessment:

The past performance citations provided by SOLERS involved much of the magnitude of effort and complexities this solicitation required. Certain projects are not related cross domain [sic] such as [NCES] Messaging and ASTG and hence the lower rating.

AR, Tab L, SRD, at 13. For BAH, the agency found that the awardee's past performance was "very relevant" based on the following assessment:

The past performance citations provided by BAH, were for efforts that involved essentially the same magnitude of effort and complexities this solicitation requires and hence relevancy rating of Very Relevant.

Id. at 14.

As to Solers' rating, DISA now concedes that the NCES and ASTG contracts were, in fact, related to cross domain requirements. DISA Response to GAO Questions at 7-8. The agency contends, however, that while the SRD states that the two contracts "are not related [to] cross domain . . . and hence the lower rating," the lower rating was not in fact based on a finding that the two contracts were not related to cross domain requirements. Id.; see AR, Tab L, SRD, at 13. Instead, the agency contends that Solers' relevancy rating was based on the agency's review of the overall requirements of the PWS. DISA Response to GAO Questions at 7-8. Specifically, DISA contends that Solers' rating was based on the agency's view that the requirements of the NCES and ASTG contracts were not of the same magnitude and complexity as the requirements of the PWS. Id.

We think that DISA's response to the protest is flatly contradicted by the contemporaneous record. In this regard, the record does not reflect that the agency considered the overall relevance of the work; instead, the evaluation specifically stated that the two contracts "are not related [to] cross domain . . . and hence the lower rating." AR, Tab L, SRD, at 13. To the extent that the agency now contends that it actually meant that the NCES and ASTF contracts did relate to cross domain requirements and that the relevancy rating was based on a different assessment, this post hoc argument does not demonstrate that the agency's evaluation was reasonable. See Boeing Sikorsky Aircraft Support, supra.

With regard to the evaluation of BAH's past performance relevancy, the contemporaneous record does not discuss which of BAH's past performance references were deemed relevant, or what aspects of those references were considered to involve the same effort as required by the RFQ. In response to the protest, the agency states that it relied on the personal knowledge of the evaluators in concluding that BAH's past performance merited a "substantial confidence rating." DISA Response to GAO Questions at 8-9. As discussed above, however, there is no discussion in the contemporaneous record concerning the use of personal knowledge by the evaluators or what that knowledge concerned.

Finally, the agency contends that the evaluators reviewed the offerors' technical proposals to determine the relevancy of their past performance, thus providing a basis to make judgments concerning past performance references for which there was no interview or PPIRS data. DISA Response to GAO Questions at 7-9. Nothing in the record supports this contention.[18] Instead, this argument was first raised by the agency in response to questions posed by our Office concerning the adequacy of the record. Our Office accords greater weight to contemporaneous source selection materials and documents and little weight to arguments raised by counsel that are not supported by the contemporaneous record. See Haworth, Inc., B-297077, B‑297077.2, Nov. 23, 2005, 2005 CPD para. 215 at 8; Chemical Demilitarization Assocs., B‑277700, Nov. 13, 1997, 98-1 CPD para. 171 at 10.

Past Performance Conclusion

In sum, we think that the record here is inadequate for our Office to determine whether DISA's evaluation of the offerors' past performance confidence and relevancy was reasonable. In this regard, the contemporaneous record does not document the information upon which the evaluators appeared to have relied in making the judgments, and the agency's response to the protest relies on numerous assertions, which are either unsupported or contradicted by the contemporaneous record. On this record, we sustain the protest.  (Solers, Inc., B-404032.3; B-404032.4, April 6, 2011) (pdf)

TCDI challenges the agency's evaluation of the firm's proposal under a number of corporate experience/past performance evaluation subfactors.

First, the protester complains that State unreasonably assessed a weakness in TCDI's proposal under the past performance subfactor, after concluding that a number of TCDI's delivered products, specifically the WMEAT database, CWC/MIMS, and the Treaty Information Portal, required significant rework and/or reengineering. Protest at 20. With respect to State's assertion that rework was required on the WMEAT database, TCDI explains that it was tasked with converting the database from an obsolete version of Oracle to the Microsoft SQL Server database system in 2007.[10] Afterwards, system tests revealed errors in the Microsoft functions and not in TCDI's work. TCDI explains that it reengineered the scripts to produce the correct output and rewrote the computer code to enable the system to work. Id. With respect to the agency's reference to rework for CWC/MIMS, TCDI states that it is unfamiliar with the acronym and speculates that it may have been a project from 10‑12 years earlier. Id. n. 9. With respect to the Treaty Information Portal, TCDI disputes that the product required significant rework, noting that the previous contracting officer's technical representative was very satisfied with the work. Id. at 20.

In reviewing protests of alleged improper evaluations and source selections, even in a task order competition as here, we do not reevaluate proposals, but rather we examine the record to determine whether the agency's judgment was reasonable and in accord with the stated evaluation criteria and applicable procurement laws and regulations. ACCESS Sys. Inc., supra. In order for us to review an agency's evaluation judgment, an agency must have adequate documentation to support its judgment. Northeast MEP Servs., Inc., B-285963.5 et al., Jan. 5, 2001, 2001 CPD para. 28 at 7. We do not limit our review to contemporaneous evidence, but consider all the information provided, including the parties' arguments, explanations, and documentation prepared in response to protest contentions. Systems Research and Applications Corp.; Booz Allen Hamilton, Inc., B-299818 et al., Sept. 6, 2007, 2008 CPD para. 28 at 12. While we consider the entire record, including the parties' later explanations and arguments, we accord greater weight to contemporaneous evaluation and source selection material than to arguments and documentation prepared in response to protest contentions. Id. Where an agency fails to provide documentation of its evaluation, it bears the risk that there may not be adequately supporting rationale in the record for us to conclude the agency had a reasonable basis for its evaluation and selection decision. Southwest Marine, Inc.; American Sys. Eng'g Corp., B-265865.3, B-265865.4, Jan. 23, 1996, 96-1 CPD para. 56 at 10.

State responds that its evaluation was based on "observed performance and user feedback" but provides no explanation or documentation of what was observed or what feedback it received. See AR at 6. Although State contends that TCDI's work on the WMEAT database was not timely completed, it does not address TCDI's explanation as to why the firm was not responsible for problems in the database. The agency also does not address in any fashion TCDI's arguments with respect to the assessed concerns with CWC/MIMS and the Treaty Information Portal. In short, State has failed to provide either contemporaneous documentation or subsequent explanation supporting its assessment of this weakness in TCDI's proposal. Accordingly, we find from the record before us that State did not have a reasonable basis for the assignment of this weakness in TCDI's proposal.

Next, TCDI challenges the agency's assessment of a weakness under the corporate experience subfactor that TCDI's proposal lacked "depth and breadth of experience relevant to working within the Federal sector" as compared to other offerors' proposals. Protest at 7-10. TCDI disagrees with the agency's assessment and contends that the RFP did not indicate that experience with other government agencies was required. Id. at 9. TCDI states that it drafted its proposal to focus on its experience with State because this is what the solicitation indicated would be evaluated. Comments at 24 n.13.

State responds that it considered this to be a minor weakness and that the RFP "clearly indicated" that the depth of work outside State would be considered because the program requires interface with interagency partners. AR at 5. However, the agency does not identify where in the RFP offerors were notified that the depth, breadth, and scope of an offeror's relevant work for other federal agencies was a requirement. Moreover, we find that the agency's evaluation record does not demonstrate that the assignment of this weakness was reasonable. State does not explain why TCDI's experience as the incumbent did not provide sufficient experience working with other agencies, such as Commerce and the Department of Defense.

TCDI also challenges State's evaluation of its proposal under the personnel qualifications/management factor, where it received a rating of green. Protest at 21. Although the TEP noted that TCDI, as the incumbent, had nearly all staff cleared and ready to immediately perform, it also noted that TCDI had failed to propose any teaming partner staff. Specifically, the TEP stated that, although TCDI's proposal stated that the firm had sufficient access to staff through its teaming partner to meet any perceived shortfalls related to certification or expertise, TCDI did not specifically offer any staffing from its teaming partner. AR, Tab 4, TEP Evaluation Report, at 5. TCDI argues that the RFP did not require offerors to identify all proposed staff, and therefore the State Department evaluated proposals on a different basis than that announced in the solicitation. Protest at 24. Moreover, TCDI contends that the agency's assignment of a weakness in this area is irrational, given that the State Department also noted a strength in TCDI's proposal under the corporate quality certifications subfactor for the various certifications of its teaming partner. Id. at 24 n.11.

State does not refute TCDI's arguments, nor does it provide an explanation for the apparent inconsistency between the weakness assigned under one evaluation factor and the strength assigned in a subfactor. Rather, State merely notes that TCDI's proposal was rated green under this evaluation factor. AR at 7.

Here too, we have no basis to find State's evaluation to be reasonable, given the agency's failure to provide any documentation or support for its evaluation. In this regard, we also agree with TCDI that the RFP did not require offerors to identify staff other than key personnel.  (Technology Concepts & Design, Inc., B-403949.2; B-403949.3, March 25, 2011  (pdf)

A1 Procurement complains that the VA improperly rejected its proposal on the basis of the firm's low price without any consideration of the firm's technical approach. In this regard, A1 Procurement states it was able to offer its low price because the firm's headquarters is only two blocks from the cemetery. Protest at 5; Protester's Proposal at 6. The protester also complains that its proposed overall price was only 6 percent below the awardee's price. Comments at 8.

The VA responds that A1 Procurement's proposed price was considered to be too low because the firm's proposed unit price of $7,200 for the trimming CLIN was considered too low in comparison to the government's unit price estimate of $16,200 for this CLIN. The agency states that on this basis it concluded that the protester's proposal was technically unacceptable. Supp. AR at 7-8.

Before awarding a fixed-price contract, an agency is required to determine that the price offered is fair and reasonable. Federal Acquisition Regulation (FAR) sect. 15.402(a). An agency's concern in making a price reasonableness determination focuses primarily on whether the offered prices are higher than warranted, as opposed to lower.[4] CSE Constr., B‑291268.2, Dec. 16, 2002, 2002 CPD para. 207 at 4. Although not required, an agency also may provide for a price realism analysis in a solicitation for the award of a fixed-price contract for the purpose of assessing whether an offeror's low price reflected its understanding of the contract requirements or the risk inherent in an offeror's approach. Id.

Here, there is no explanation in the record as to the purpose for the agency's evaluation of whether an offeror's proposed fixed price was too low. The RFP does not state that the VA intended to perform a price realism analysis to assess the offerors' understanding or to assess performance risk, nor does the VA assert that it performed a price realism analysis.

Regardless of the agency's purpose for assessing whether offerors' proposed fixed prices were too low, however, we conclude that the VA's assessment that A1 Procurement's price was too low is unsupported by the record and does not appear to be consistent with the solicitation. As noted above, the RFP provided that the agency would consider, among other things, the offerors' price breakdowns, pricing methodology, and technical approaches and capabilities in assessing whether proposed prices were too low or too high. RFP at 90. There is no documentation in the record (nor does the agency claim) that such an analysis was performed with respect to the protester's proposal. Rather, the record contains only the VA's conclusory judgment that the protester's fixed price was too low. An agency's evaluation must be sufficiently documented to allow review of the merits of a protest. Northeast MEP Servs., Inc., B-285963.5 et al., Jan 5, 2001, 2001 CPD para. 28 at 7. Where an agency fails to provide documentation of its evaluation, it bears the risk that there may not be adequately supporting rationale in the record for us to conclude the agency had a reasonable basis for the selection decision. Id.

Furthermore, here the VA initially informed the protester that its price was too low and represented a performance risk because A1 Procurement's fixed price was significantly below the GCE and the prices of the other offerors. See AR, Tab 6, VA Debriefing Letter, Dec. 1, 2010. In response, the protester noted that its price was only 6 percent below the awardee's price. Rather than explain why it viewed the protester's overall fixed price to be too low, the VA asserted that A1 Procurement's proposal was rejected because the agency found the firm's proposed price for one of the five CLINs to be too low. See Supp. AR at 7-8. There is no contemporaneous documentation evidencing the agency's concern that the protester's price for one CLIN was too low in comparison to the GCE for this CLIN. Nor does the VA state how the protester's price compares to the awardee's or other offerors' prices for this CLIN or overall. There is also no explanation in the record as to why A1 Procurement's low price for this one CLIN reflects a performance risk.

In short, the record is devoid of contemporaneous documentation showing that the agency evaluated A1 Procurement's price in accordance with the RFP, and contains no explanation in response to the protest to support the agency's conclusion that A1 Procurement's price is too low. Accordingly, we find unreasonable the VA's rejection of A1 Procurement's proposal on the basis of the agency's unsupported judgment that the firm's price was too low.  (A1 Procurement, JVG, B-404618, March 14, 2011)  (pdf)

NEMS contends, among other things, that the agency ignored adverse past performance information concerning the awardee's performance at a number of DeCA commissaries. See Protest at 5; Comments at 9-12; Protester's Hearing Comments at 6.

The record here shows that the technical evaluation team received past performance information from DeCA commissaries and from two other federal agencies. See AR, Tab 9, Past Performance Information, at 92-185. Although this information included a number of favorable comments regarding Nayyarsons' past performance, e-mails from several DeCA commissaries also indicated that Nayyarsons had a number of performance problems. Specifically, the evaluation team was provided with e-mails from various commissaries which reported staff and product shortages during transition periods; unsanitary conditions; employee tardiness and cleanliness; and problems with sushi, including use of expired products and pre-dating products. See, e.g., id. at 147-48 (June 25 e-mail reporting "serious problems" with sushi at Memphis commissary which "nee[d] to be addressed immediately"); 145-46 (July 15 e-mails reporting an "ongoing problem" and that "there are still significant issues" with sushi at Memphis commissary); 154 (July 18 e-mail reporting problems "once again" with sushi at Scott AFB).

As a general matter, the evaluation of an offeror's past performance is within the discretion of the contracting agency, and we will not substitute our judgment for reasonably based past performance ratings. See MFM Lamey Group, LLC, B‑402377, Mar. 25, 2010, 2010 CPD para. 81 at 10. While we have recognized that such judgments are often subjective by nature, the exercise of these judgments in the evaluation of proposals must be documented in sufficient detail to show that they are not arbitrary. That is to say, in order for us to review an agency's evaluation of proposals, an agency must have adequate documentation to support its judgment. Northeast MEP Servs., Inc., B‑285963.5 et al., Jan. 5, 2001, 2001 CPD para. 28 at 7. See American President Lines, Ltd., B-236834.3, July 20, 1990, 90-2 CPD para. 53 at 6.

Where a protester challenges the past performance evaluation and source selection, we will review the evaluation and award decision to determine if they were reasonable and consistent with the solicitation's evaluation scheme and procurement statutes and regulations, and to ensure that the agency adequately documented the basis for the selection. Wackenhut Servs., Inc., B-400240, B‑400240.2, Sept. 10, 2008, 2008 CPD para. 184 at 6; S4, Inc., B-299817, B‑299817.2, Aug. 23, 2007, 2007 CPD para. 164 at 9. When an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for its source selection decision. Southwest Marine, Inc.; American Sys. Eng'g Corp., B‑265865.3, B-265865.4, Jan. 23, 1996, 96-1 CPD para. 56 at 10.

The contemporaneous record here shows no consideration by the technical evaluation team or the CO of the awardee's recent performance problems identified in these e‑mails. Rather, the evaluation team rated Nayyarsons' proposal "exceptional" for each of the two past performance subfactors, despite the team's receipt of numerous e-mail reports of problems with Nayyarsons' performance. See AR, Tab 9, Past Performance Information, at 89‑92. Moreover, the record shows that several of the e‑mails identifying problems with Nayyarsons' performance were originally addressed to the CO here, and, during the hearing, the CO testified that she reviewed all of the information that was provided to the evaluation team. See, e.g., id. at 132, 145, 148, 150-54 (e-mail reports of awardee's adverse past performance addressed to CO); Tr. at 74.

In our view, Nayyarsons' past performance rating is inconsistent with the agency's stated rating scheme, which provided that an "exceptional" rating would only be assigned for past performance reflecting few minor problems for which corrective actions taken by the contractor were highly effective. As discussed above, nothing in contemporaneous record shows that the agency considered whether the firm in fact took effective steps to correct the performance problems reported by DeCA commissaries. See AR, Tab 9, Evaluators' Notes, at 75-91; Tab 5, Decision Summary, at 1-23. For example, the record shows that the Memphis commissary reported serious problems with Nayyarsons' sushi products on June 25, but that significant problems with sushi were ongoing as of July 15. The record also shows that on July 18, the commissary at Scott AFB reported renewed problems with Nayyarsons' sushi products.

Based on this record, we conclude that the agency ignored adverse past performance information and--in assigning a past performance rating of exceptional to the awardee--deviated from the agency's evaluation scheme. See G. Marine Diesel, B‑232619.3, Aug. 3, 1989, 89-2 CPD para. 101 at 6‑7 (protest sustained where agency ignored awardee's performance problems); Apptis, Inc., B‑299457, et al., May 23, 2007, 2008 CPD para. 49 at 11 (evaluation cannot be determined to be reasonable where record lacks documentation that agency considered deficiencies in awardee's proposal); Midland Supply, Inc., B‑298720.3, May 14, 2007, 2007 CPD para. 104 at 5-6 (past performance evaluation unreasonable and inconsistent with awardee's record and agency's rating scheme); International Bus. Sys., Inc., B‑275554, Mar. 3, 1997, 97‑1 CPD para. 114 at 5 (past performance information too close at hand for agency to fail to consider the information). Accordingly, we sustain the protest on this basis.  (Northeast Military Sales, Inc., B-404153,January 13, 2011)  (pdf)

Contrack argues that the agency's assessment of Zafer's past performance as excellent was unreasonable because the Corps failed to consider adverse past performance information concerning Zafer's past performance. Specifically, the protester complains that the agency did not consider negative CPARs for three of Zafer's projects, one of which was a contract with the Corps for the renovation and repair of the National Army Military Hospital in Kabul, Afghanistan. This contract was the subject of the DoD IG report that Contrack provided to the Corps with its initial protest.

Contrack also complains that the agency ignored the news article (which was provided with the initial protest) that reported that the DoD IG would be investigating other deficiencies found by the Commission on Wartime Contracting in Iraq and Afghanistan with respect to Zafer's performance of another Corps contract for construction services related to a headquarters building in Kabul.[4] Protest at 11. Contrack contends that the agency, in its review of the CPARs, unreasonably assigned Zafer's proposal an excellent rating based upon the company's performance of two smaller projects, while ignoring negative reports for the three larger projects. Comments at 7.

The Corps responds that Zafer's excellent past performance rating was based upon the "totality of the past performance information," which the Corps argues was "laudatory." See CO's Statement at 2. In this regard, the SSA states that one of the projects upon which the Corps based its overall past performance rating was for an ammunition supply point, which the Corps states was more directly relevant than Zafer's other construction projects. Finally, the Corps argues that it had no obligation to consider the adverse information about Zafer's prior performance contained in the DoD IG report, any materials developed by the Wartime Contracting Commission, or the news report concerning Zafer's past performance. In the agency's view, the Corps acted reasonably in ignoring these "external materials," even though the agency acknowledges that these materials were provided to it by Contrack as part of the earlier protest filings. AR at 5-6.

As a general matter, the evaluation of an offeror's past performance is within the discretion of the contracting agency, and we will not substitute our judgment for reasonably based past performance ratings. However, we will question an agency's evaluation conclusions where they are unreasonable or undocumented. Clean Harbors Envtl. Servs., Inc., B-296176.2, Dec. 9, 2005, 2005 CPD para. 222 at 3; OSI Collection Servs., Inc., B-286597, B-286597.2, Jan. 17, 2001, 2001 CPD para. 18 at 6. The critical question is whether the evaluation was conducted fairly, reasonably, and in accordance with the solicitation's evaluation scheme, and whether it was based on relevant information sufficient to make a reasonable determination of the offeror's past performance, including relevant information close at hand or known by the contracting personnel awarding the contract. OSI Collection Servs., Inc., supra, at 6. As explained below, we find that the agency's past performance evaluation did not meet this standard.

The record shows that Zafer's excellent past performance rating was primarily based upon two outstanding CCASS performance ratings for projects in Afghanistan, which the SSA found to be similar to the current requirement. See Original CO's Statement, Oct. 2, 2009, at 2. The original SSEB noted as strengths in Zafer's proposal that Zafer had provided many outstanding letters of commendation, that there were minimal field changes and requests for equitable adjustments, that Zafer had outstanding ratings on projects in Afghanistan, that Zafer had a good safety record, and that many of Zafer's projects were completed ahead of schedule. See AR (B‑401871), Tab 8, Original SSEB Report, at 18-19.

As noted by Contrack during its initial protest, Zafer's initial proposal identified nine projects for its experience and, in accordance with the RFP requirements, provided past performance forms for all nine projects. See Protest at 11. The Corps obtained CPARs for five of Zafer's identified projects (all of which were performed in Afghanistan), which the Corps provided with its report on Contrack's initial protest. Three of these reports, however, provided overall ratings of satisfactory or marginal, and indicated that Zafer had a number of performance problems on these projects. For example with respect to the CPAR for Zafer's National Army Military Hospital project in Kabul (overall value of $18.9 million), the report indicated that Zafer's performance rating was marginal overall; the report also contained such remarks as "[d]eficient work was not corrected in a timely [manner]," "multiple notices of deficiency were issued with no resolution," and "[p]erformance was consistently poor without correction." This was the project that was the subject of the DoD IG report that Contrack provided to the Corps with its initial protest. See Protester's Comments, exh. 5, CPAR for Contract No. W917PM-05-C-0005. Similarly, the CPAR for another of Zafer's reported projects, the design and build of a hospital facility at Baghram Airfield, indicated that Zafer's performance received an overall satisfactory rating, but noted that some of its work (including the quality of its work) was assessed as marginal.

Although the Corps generally argues that it based its assessment of Zafer's past performance as excellent upon the "totality" of Zafer's record, there is no evidence in the contemporaneous evaluation record of the agency's consideration of these three CPARs, which were identified by the protester. Moreover, the Corps has not shown in response to the protest how it could reasonably conclude that Zafer's overall past performance was excellent in light of the three CPARs indicating only satisfactory or marginal performance on a number of construction projects in Afghanistan. In this regard, we recognize that the Corps may not have been aware, at the time of its initial evaluation and selection decision, of the DoD IG report and other information that may indicate negative past performance by Zafer, but the Corps was provided with this information during the earlier protest. Nonetheless, there is no indication in the record that the Corps made any effort to investigate the merits of these reports, at least one of which appears to be documented in one of the CPARs available to the agency. In fact, it appears from the record that the newly‑appointed SSEB only reviewed Zafer's revised proposal under the past performance factor to see if the firm addressed the single weakness identified during discussions. See AR, Tab 10, SSEB Report, Jan. 26, 2010, at 3; see also Tab 10, SSEB Consensus Evaluation Worksheet for Zafer, Past Performance Factor ("no increase in rating – already Excellent").

In the absence of evidence in the record that the Corps considered relevant information available to it concerning Zafer's past performance, we have no basis to find reasonable the agency's assessment of an excellent rating for Zafer under the past performance factor. We sustain Contrack's protest on this basis. See GTS Duratek, Inc., B-280511.2, B-280511.3, Oct. 19, 1998, 98-2 CPD para. 130 at 14 (protest sustained where agency ignored known past performance information).  (Contrack International, Inc., B-401871.5, B-401871.6, B-401871.7, May 24, 2010)  (pdf)

The ATO and DEA protests raise numerous issues regarding the Navy's public-private competition and the selection of PMI. The protesters argue, among other things, that the Navy's evaluation of PMI's technical and price proposals was unreasonable, that the agency's discussions with the ATO were not meaningful, and that the Navy's corrective action improperly limited the aspects of proposals that offerors could revise. The protesters also argue that the contract award to PMI is improper because the Navy failed to complete the public-private competition study here within the required 30-month statutory timeframe regarding the expenditure of appropriated funds for such studies.

As detailed below, we find the agency's evaluation of PMI's staffing plan--a protest ground raised only by the ATO--was improper. Although we do not specifically address the protesters' remaining challenges to the Navy's evaluation of proposals, we have fully considered all of them and find that they are without merit.

An agency that fails to adequately document its evaluation of proposals bears the risk that its determinations will be considered unsupported, and absent such support, our Office may be unable to determine whether the agency had a reasonable basis for its determinations. Rosemary Livingston--Agency Tender Official, B-401102.2, July 6, 2009, 2009 CPD para. 135 at 10, recon. denied, Department of the Navy--Request for Modification of Remedy, B-401102.3, Aug. 6, 2009, 2009 CPD para. 162 at 4; Rhonda Podojil--Agency Tender Official, B-311310, May 9, 2008, 2008 CPD para. 94 at 4; Matrix Int'l Logistics, Inc., B‑272388.2, Dec. 9, 1996, 97-2 CPD para. 89 at 5.

In this case, as explained above, the Navy's original staffing estimate for the mobile refueling fuel sampling and testing requirement (Spec Item 3.3.1) was predicated on the historical number of samples completed (5,824) and an estimated 40 minutes per occurrence, which (together with overtime) resulted in a total of 3,946 labor hours. When the Navy later changed its estimate to 30 minutes per occurrence, the Navy then concluded that 2,956 total labor hours were required to adequately perform mobile refueler fuel sampling and testing requirement.

The TEB determined that PMI's original proposal with regard to the staffing of Spec Item 3.3.1 was deficient, insofar as the 9.9 minutes per fuel sampling and testing occurrence was too low. PMI's response to discussions revised the fuel sampling and testing time upward to 15.88 minutes, but the offeror also reduced the number of fuel sampling and testing occurrences to 2,720. However, PMI's proposal did not explain how it had determined that the PWS required only 2,720 fuel samples annually,[11] or how it had determined that 15.88 minutes per occurrence was adequate to perform the requirements; it also appears that the offeror "backed into" its fuel sampling time by first allocating the total number of labor hours.

The TEB subsequently concluded that PMI's revised proposal had remedied all deficiencies and that its proposed staffing for Spec Item 3.3.1 (and ELIN 005) was adequate. However, the record does not indicate why it was reasonable for the TEB to conclude that PMI's proposed time of 15.88 minutes per fuel sample test was adequate, when it differed substantially from the Navy's time estimate. The record also does not indicate why it was reasonable for the TEB to conclude that PMI's assumption that only 2,720 fuel test samples per year were required was acceptable, when it differed substantially from the agency's IGE and the historical number of 5,824 annual fuel test samples. As a result, the record does not support the TEB's conclusion that PMI's proposed 960 labor hours were sufficient to perform Spec Item 3.3.1, given that the Navy believed an offeror's staffing had to be based on at least 2,956 total hours.

Further, the record shows that when the ATO attempted to revise its staffing for Spec Item 3.3.1 by using a 15-minute sampling time, the TEB expressly disagreed and found that it made the ATO's proposal technically unacceptable.[12] In this regard, while the TEB found that the ATO's proposal had failed to adequately support how the MEO would accomplish the fuel tests in 15 minutes per sample, the record shows that PMI's proposal--which the TEB found acceptable--likewise failed to provide any information to support how it would accomplish the same tests in 15.88 minutes per sample, referring only to the increase from its original, lower time per sample. Similarly, with regard to the number of fuel samples required, the record shows that while the TEB found that PMI's assumption and the corresponding staffing were acceptable, the TEB used the higher IGE number (5,824 samples) and corresponding staffing when assessing the adequacy of the ATO's proposed staffing for the requirement. In sum, while the TEB evaluated the ATO's proposal based on a required minimum of 2,956 labor hours for Spec Item 3.3.1 (and concluded that the 1,529 labor hours proposed by the ATO were unacceptable), the TEB concluded without explanation that the 960 labor hours proposed by PMI for the same task were sufficient. There is no explanation in the record for the inconsistencies in the agency's evaluation of PMI's and the ATO's proposals in this area.

The agency argues that it reasonably determined the adequacy of PMI's staffing for all of ELIN 0005, and that it was at the ELIN-level (and not the underlying Spec Item-level) that staffing adequacy was measured. The record does not support this argument. As a preliminary matter, the record reflects that the IGE had staffing estimates for each Spec Item, and that the ELIN staffing estimates were based on the Spec Item staffing estimates. The record also reflects that the Navy assessed the adequacy of offerors' staffing plans at both the Spec Item and ELIN levels. Moreover, when determining that the ATO's staffing plan for Spec Item 3.3.1 was insufficient, the TEB's evaluation did not extend beyond the one specific task and consider the offeror's staffing for the entire ELIN. The agency also found PMI's original staffing for Spec Item 3.3.1 to be a deficiency, and there is nothing in the record to suggest that it was remedied by PMI's staffing of other aspects of ELIN 0005.

The Navy also argues that the evaluation record was adequate to support its determination that PMI's staffing for Spec Item 3.3.1 was acceptable; the agency essentially argues that "not much need be said" when determining a proposal meets (as opposed to failing to meet) the requirements. We disagree. An agency's evaluation of proposals must be adequately documented in order to establish the reasonableness of its determinations. See Urban-Meridian Joint Venture, B-287168, B-287168.2, May 7, 2001, 2001 CPD para. 91 at 2. This requirement applies equally to evaluation determinations of proposal acceptability and determinations of proposal unacceptability, weakness, or deficiency. Here, the TEB concluded without explanation or other support that PMI's revised proposal had adequately staffed Spec Item 3.3.1 notwithstanding the fact that the underlying number of fuel sample occurrences, time per occurrence, and labor hours, differed materially from the Navy's estimates. To the extent some other aspect of PMI's proposal made its staffing for the mobile refueler fuel sampling and testing requirement acceptable (e.g., cross-utilization of labor from other Spec Items), such analysis is not part of the TEB's conclusory determination of PMI's staffing adequacy.

Given the inadequate documentation in the record before us to support a key finding that PMI's staffing plan was acceptable, we sustain the protest on this basis.  (Bruce Bancroft--Agency Tender Official; Sam Rodriquez--Designated Employee Agent, B-400404.7; B-400404.8; B-400404.9; B-400404.10; B-400404.11, November 17, 2009)  (pdf)

Our Office will review the documentation supporting a source selection decision to determine whether that decision was adequately supported and rationally related to the evaluation factors. Universal Shipping Co., B-223905.2, Apr. 20, 1987, 87-1 CPD para. 424, at 10.

Implicit in the foregoing is that the evaluation must be documented in sufficient detail to show that it was not arbitrary, Adelaide Blomfield Mgmt. Co., B-253128.2, Sept. 27, 1993, 93-2 CPD para. 197 at 4, and a selection decision may not be made on point scores alone where the agency selection official has inadequate documentation on which to base a reasoned decision. J.A. Jones Mgmt. Servs., Inc., B‑276864, July 24, 1997, 97-2 CPD para. 47 at 4. While we are mindful that when an agency places an order under a BPA, limited documentation of the source selection is permissible, the agency must at least provide a sufficient record to show that the source selection was reasonable. FAR sect. 13.303-5(e); see also FAR sections 13.106‑3(b)(3)(ii) (in a simplified acquisition record must include additional statements to explain the basis for award when factors other than price are used).

In short, the contemporaneous record here is inadequate for our Office to determine whether the Forest Service had a reasonable basis to select Aquatic at its higher price. Additionally, the record shows that the Forest service used an evaluation factor (equipment) that was not disclosed in the solicitation, and which had the effect of reducing the significance of the cost/price factor.[7] Since the record provides no contemporaneous tradeoff comparing Aquatic to C&B, other than on the basis of their point scores, we sustain the protest. See. Shumaker Trucking & Excavating Contractors, Inc., B-290732, Sept. 25, 2002, 2002 CPD para. 169 at 8 (protest sustained where Forest Service relied solely on point scores and failed to document any comparison of protester's lower-priced and lower-rated proposal to awardee's higher-priced, higher-rated proposal, in source selection decision).

With respect to the Forest Service's more detailed explanation of its evaluation, offered in response to C&B's protests, our Office generally gives little weight to such reevaluations and judgments prepared in the heat of the adversarial process. Boeing Sikorsky Aircraft Support, B‑277263.2, B‑277263.3, Sept. 29, 1997, 97‑2 CPD para. 91 at 15. In our view, the explanation offered by the Forest Service in response to the protests cannot overcome the failure of the contemporaneous record to provide any explanation of the evaluation of C&B (other than point scores) on which the agency could make a reasoned selection decision.  (C&B Construction, Inc., B-401988.2, January 6, 2010) (pdf)

Finally, GAI protests that the agency’s evaluation and source selection decision was inadequately documented, complaining that “[n]o evaluation worksheets, individual evaluator notes or scales were produced.” GAI Comments and Supplemental Protest, Mar. 5, 2009, at 3. GAI’s protest in this regard is without merit.

Although an agency must document its evaluation judgments in sufficient detail to show that they are not arbitrary, the necessary amount and level of detail will vary from procurement to procurement. U.S. Defense Sys., Inc., B-245563, Jan. 17, 1992, 92-1 CPD para. 89 at 3; Champion-Alliance, Inc., B-249504, Dec. 1, 1992, 92-2 CPD para. 386 at 6-7. For example, there is no requirement that the evaluation record must include narrative explanations for every rating assigned. Apex Marine Ship Mgmt. Co., LLC; American V-Ships Marine, Ltd., B-278276.25, B-278276.28, Sept. 25, 2000, 2000 CPD para. 164 at 8-9. Similarly, there is no requirement that an agency retain individual evaluator’s notes or worksheets, provided the agency’s final evaluation documentation reasonably explains the basis for the agency’s judgments. Global Eng’g and Constr. LLC, B-290288.3, B‑290288.4, Apr. 3, 2003, 2003 CPD para. 180 at 3 n.3.

Here, the contemporaneous record included a detailed technical evaluation report that included the agency’s narrative explanation regarding the basis for each evaluation rating of either “Unacceptable” or “Outstanding.” AR, Tab 32. Further, the narrative explanations supporting the ratings consistently include specific references to the particular portions of the offerors’ proposals that formed the basis for the agency’s assessments. Id. Finally, the agency’s source selection documentation contained a detailed comparative discussion of the two offerors’ proposals, identifying particular distinguishing features of each proposal. AR, Tab 34. On this record there is no merit in GAI’s assertion that the agency’s evaluation and source selection decision were inadequately documented.  (Government Acquisitions, Inc., B-401048; B-401048.2; B-401048.3, May 4, 2009)  (pdf)

Here, the record is inadequate to establish that the contracting officer’s finding of technical equality is reasonable and proper. As discussed above, the evaluation record consists of the evaluators’ adjectival ratings for each of the subcriteria, their narrative comments under several subcriteria, and the contracting officer’s scoring of the proposals based on the adjectival ratings. The evaluators did not provide the contracting officer with a comprehensive assessment or listing of the proposals’ strengths and weaknesses, and the record includes no evidence that the contracting officer ever considered the actual merits of the proposals in calculating the scores. Likewise, there is no indication that the contracting officer considered the actual merits of the proposals in ultimately determining that, notwithstanding ROG’s proposal’s approximately 12 percent higher score, it was equal in technical merit to RPCI’s proposal. The record includes no explanation of the contracting officer’s rationale for her conclusion that the approximate 12 percent scoring difference did not translate into actual technical superiority for RPG’s proposal. Rather, the record includes only the conclusory statement that “After performing the evaluation, it was determined that the offers were equal both technically and in past performance … .” AR exh. 15, Price Negotiation Memorandum, Aug. 23, 2007. This brief statement is the sole contemporaneous explanation for the contracting officer’s determination that the proposals were technically equal, notwithstanding ROG’s proposal’s higher percentage score.

In reviewing an agency’s evaluation, we may also consider documentation prepared after the source selection decision was made, although we will accord greater weight to contemporaneous materials rather than judgments made in response to protest contentions. Beacon Auto Parts, B-287483, June 13, 2001, 2001 CPD para. 116 at 6. Here, the agency submitted no contracting officer’s statement in response to the protester’s supplemental protest, in which ROG’s specific evaluation challenges are raised, and the post-protest record, like the contemporaneous record, contains no other support for the contracting officer’s conclusion that the proposals were technically equal. In a memorandum dated October 11, 2007, prepared after the filing of RPCI’s prior protest, the contracting officer concluded that “The combined scoring for Technical and Past Performance was found to be equal.” AR exh. 14, Contracting Officer Memorandum, Oct. 11, 2007. Similarly, in the narrative submitted in response to ROG’s initial protest, the contracting officer merely recites that she “determined that the offerors were essentially technically equal based on the technical and past performance factors set forth in the RFP.” AR exh. 1, Contracting Officer’s Narrative, July 7, 2008, at 1. These conclusory statements are inadequate to establish the reasonableness of the contracting officer’s determination that the proposals were technically equal.

While the agency’s report in response to ROG’s supplemental protest does not include a statement by the contracting officer, it does respond to each of the protester’s specific challenges to the evaluation ratings. However, these responses were provided by the agency’s legal counsel, with no indication that the responses reflect the contracting officer’s own rationale for her evaluation conclusions. In this regard, the agency’s counsel provides explanations for the various challenged ratings but, instead of attributing the asserted rationales to the contracting officer, asserts that the explanations would lead “a reasonable person” to conclude that the scoring was reasonable. Supp. AR at 5-7. These responses do not constitute an adequate evaluation record, see York Bldg. Servs., Inc., B‑296948.2 et al., Nov. 3, 2005, 2005 CPD para. 202 at 7 (GAO accords little or no weight to “new rationales, based on a hypothetically correct evaluation, for which there is no support in the contemporaneous record.”), and the supplemental report does not otherwise indicate the considerations that factored into the contracting officer’s determination that the proposals were technically equal.

We conclude that the contracting officer’s determination that ROG’s and RPCI’s proposals were technically equal lacked adequate supporting explanation or documentation and, therefore, was unreasonable. See Magellan Health Servs., B‑298912, Jan. 5, 2007, 2007 CPD para. 81 (protest challenging adequacy of agency’s source selection decision sustained where evaluation record was insufficient to establish reasonableness of the selection official’s determination that offers were technically equal, notwithstanding protester’s proposal’s higher technical rating); Midland Supply, Inc., B‑298720, B‑298720.2, Nov. 29, 2006, 2007 CPD para. 2 (award decision not reasonable where there is no documentation or explanation and agency makes its award decision based strictly on a mechanical comparison of the offerors’ total point scores)Accordingly, we sustain the protest on this ground.  Radiation Oncology Group of WNY, PC, B-310354.2; B-310354.3, September 18, 2008) (pdf)

The agency tender official asserts that, after finding the agency tender "acceptable" after several rounds of discussions, the agency improperly conducted yet another round of discussions, as a result of which the agency tender no longer was the lowest-priced tender/offer received. In response, the agency argues that the TEB in fact did not find the agency tender acceptable until after the final round of discussions, despite having described the agency tender as "acceptable" in the TEB report prepared after the prior round of discussions. The key question in resolving the protest thus is whether the agency's evaluation record adequately shows that the agency did--or did not--find the agency tender acceptable before the final round of discussions. As discussed below, we think the record is inconclusive in this regard.

Although the FAR does not specify what is required to be documented in the contract file in support of an agency’s evaluation of proposals, see FAR sect. 15.305(a), the fundamental principle of government accountability dictates that an agency maintain a record adequate to allow for the meaningful review of the merits of a protest. This principle applies equally in the context of a public-private competition under the Circular. See Rhonda Podojil--Agency Tender Official, B-311310, May 9, 2008, 2008 CPD para. 94 at 4. An agency that fails to adequately document its evaluation of proposals bears the risk that its determinations will be considered unsupported, and absent such support, our Office may be unable to determine whether the agency had a reasonable basis for its determinations. Matrix Int’l Logistics, Inc., B‑272388.2, Dec. 9, 1996, 97-2 CPD para. 89 at 5. That is not to say that our Office, in determining the reasonableness of an agency’s evaluation and award decision, limits its review to the contemporaneous evaluation and source selection documentation. Rather, we will consider, in addition to the contemporaneous documentation, all information provided to our Office for consideration during the protest, including the parties' arguments and explanations, so long as the information is credible and consistent with the contemporaneous record. Id.; NWT, Inc.; PharmChem Labs., Inc., B-280988, B-280988.2, Dec. 17, 1998, 98-2 CPD para. 158 at 16.

In considering the entire record, we accord greater weight to contemporaneous evaluation and source selection material than to the parties' later explanations, arguments, and testimony. Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD para. 91 at 15. Where the record before us is inconsistent or incomplete to such an extent that we cannot find the agency's evaluation of proposals to be reasonable, we will sustain the protest. Carahsoft Tech. Corp.; Allied Tech. Group, B-311241, B-311241.2, May 16, 2008, 2008 CPD para. 119 at 8-9 (sustaining protest where the record contained inadequate documentation to show the reasonableness of the agency's evaluation, and the agency's arguments appeared inconsistent with the contemporaneous record); Honeywell Tech. Solutions, Inc.; Wyle Labs., Inc., B‑292354, B-292388, Sept. 2, 2003, 2005 CPD para. 107 at 7 (sustaining protest where "we simply cannot determine from [the] record which aspect of the agency’s evaluation was reasonable and which was unreasonable").

In this case, as explained above, the agency held a total of five rounds of discussions with the agency tender official; at the conclusion of the fourth round, the agency tender was lower-priced than the eventual awardee's offer. Thus, if (as the agency tender official argues) the agency tender in fact was found acceptable after the fourth round of discussions, holding the fifth and final round of discussions was improper. Further, those discussions resulted in prejudice to the agency tender because the agency tender's price increased as a result, to a price greater than the eventual awardee's, thus displacing the agency tender as the lowest-priced technically acceptable offer/tender. In response, the agency argues that the record clearly shows that, despite having described the agency tender as "acceptable" before initiating the fifth round of discussions, the TEB in fact found the agency tender "unacceptable," and thus properly conducted another round of discussions.

As discussed below, we think that the evaluation record is inconsistent and inconclusive with respect to the TEB's findings regarding the agency tender after the fourth round of discussions. As a result, we conclude that the record here is inadequate to support a conclusion that the decision to hold the final round of discussions was proper. Accordingly, we sustain the protest on this basis.

The dispute regarding the TEB's findings derives principally from the conflicting language in the July TEB report, specifically, the repeated description of the agency tender as "acceptable" alongside this sentence: "This weakness will need to be corrected before implementing the MEO can be considered." The agency does not assert that these statements are reconcilable; rather, the agency argues that the TEB made an error in describing the agency tender as "acceptable" and that the quoted sentence from the TEB report is contemporaneous evidence that the TEB in fact considered the fourth revised agency tender "unacceptable." In support of its position, the agency, pointing to the RFP definitions of "acceptable" and "marginal" proposals, asserts that the agency tender could not be regarded as "acceptable" because it contained a weakness that had to be corrected before implementation of the MEO.

Even accepting the agency's interpretation of the RFP definitions of "acceptable" and "marginal," the agency's argument still does not resolve the conflict between the TEB report's description of the agency tender as "acceptable" and the finding in the same report that the agency tender had a weakness needing correction before the MEO could be considered. Those two statements cannot both be accurate, and we see no basis to conclude that the "mistake" was in the characterization of the agency tender as "acceptable," rather than in the finding that the agency tender contained a deficiency, as evidenced by the statement that the agency tender contained a weakness requiring correction. On the contrary, the record strongly suggests that the description of the agency tender as "acceptable" reflects a deliberate choice by the TEB, given that, in the prior three rounds of discussions and evaluations, the agency characterized the agency tender as "poor" before making the significant change--to repeatedly describing the agency tender as "acceptable," sometimes in bold capital letters--in the July report.

The agency also argues that the discussions letter to the agency tender official identifying deficiencies in the tender, which stated that, "[a]lthough your tender remains in the competitive range, it continues to contain deficiencies," is contemporaneous evidence that the TEB had found the then-most recent tender to be unacceptable. We are not persuaded that this letter reasonably can be regarded as reflecting a determination by the agency that the agency tender was "unacceptable." As noted above, the language of that letter is essentially identical to the language used in the prior discussions letter, suggesting that the agency may have copied the "boilerplate" portion of the prior letter rather made a deliberate language choice. Similarly, the letter uses the plural term--"deficiencies"--to describe the agency tender, while the TEB Report at most identified a single deficiency in the agency tender, further suggesting that the letter does not warrant the dispositive weight urged by the agency.

Finally, the agency argues that, notwithstanding that the agency tender was mistakenly labeled "acceptable" in the July 29, 2008 TEB report, it is otherwise clear from the record that the fourth revised agency tender failed to meet a material term of the RFP and therefore was not acceptable. As noted above, that TEB report stated that the agency tender had the “appropriate amount of [DELETED] FTEs to handle [DELETED]; however, the TEB is still concerned that the [DELETED] FTEs will not be able to handle [DELETED],” AR, Exh. 31, TEB Report of July 29, 2008 at 25, and it was not until its response to the agency tender official’s comments on the agency report that the agency first alleged that the revised agency tender failed to meet a government requirement. The TEB found that the agency tender had met the more time-intensive requirement of the RFP with respect to [DELETED], that is, it had staffed [DELETED]. Now the TEB was left to determine whether the agency tender had allotted sufficient hours [DELETED]. The TEB report recounted in some detail the rationale that the agency tender advanced in support of its assertion that it had allotted sufficient hours, without concluding that the agency tender was inadequate in any way. A full reading of the TEB report suggests that the TEB was identifying a concern about the hours allocated [DELETED], but that, given the approach to the requirement proposed in the agency tender, the agency tender was no longer unacceptable or deficient. Absent a clear statement that the fourth revised tender failed to meet a material requirement of the RFP--statements that the agency previously had made repeatedly throughout the competition--on this record we do not think it is reasonable to infer such a finding, particularly where doing so would contradict the agency's own explicit statement that the tender was "acceptable."

On the record before us, with inconsistent statements by the agency in its evaluation of the fourth revised tender and inadequate documentation of a finding that the tender was unacceptable, we sustain the protest.  (Rosemary Livingston--Agency Tender Official, B-401102.2, July 6, 2009)  (pdf)  See (Department of the Navy--Request for Modification of Remedy, B-401102.3, August 6, 2009)  (pdf)

MINACT asserts that the evaluators' contemporaneous reasons for not increasing its score are invalid and fail to provide a rational basis for the evaluation. TRESP Assocs, Inc.; Advanced Data Concepts, B‑258322.5, B‑258322.6, Mar. 9, 1995, 96‑1 CPD para. 8 at 4. It further asserts that the evaluators' hearing explanations are both irrational and represent post hoc rationalizations, to be accorded little, if any weight. Post Hearing Comments at 27; see Boeing Sikorsky Aircraft Support, B‑277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD para. 91 at 15.

An agency is required to document its judgments in sufficient detail to show that they are not arbitrary. TRESP Assocs, Inc.; Advanced Data Concepts, supra. However, in reviewing an agency's evaluation, we do not limit our review to contemporaneous evidence, but consider all of the information provided, including the parties' arguments, explanations, and any hearing testimony. Remington Arms Co., Inc., B-297374, B‑297374.2, Jan. 12, 2006, 2006 CPD para. 32 at 10. While we generally give little weight to reevaluations and judgments prepared in the heat of the adversarial process, Boeing Sikorsky Aircraft Support, supra, post‑protest explanations that provide a detailed rationale for contemporaneous conclusions simply fill in previously unrecorded details, and will generally be considered in our review of the reasonableness of an evaluation, so long as those explanations are credible and consistent with the contemporaneous record. NWT, Inc.; PharmChem Labs., Inc., B‑280988, B-280988.2, Dec. 17, 1998, 98-2 CPD para. 158 at 16.

The evaluators' explanations are sufficient to establish a reasonable basis for their evaluations. In this regard, MINACT's argument is based upon a faulty assumption--that its proposal was downgraded based on the assessed weaknesses and that, when the weaknesses were corrected, its proposal score should have been increased by all remaining points--in essence, that it should have received a perfect score. Post Hearing Comments at 28; exh. 27, at 15. There is no basis for this assumption. First, nothing in the RFP indicated that more than a satisfactory score would be assigned proposals meeting the RFP requirements. Absent such an RFP provision, we think it is indisputably reasonable for a proposal to be rated with an adjective of fully satisfactory--rather than very good or exceptional--where the correction of weaknesses results in the proposal's meeting--rather than exceeding--the RFP requirements. This was the approach followed by the evaluators. Tr. at 50, 151, 209. Since, as explained by the evaluators, MINACT's improvements merely brought its proposal up to the expected and satisfactory level, as set forth in the RFP, and were found to provide no value beyond that level, the evaluators could reasonably conclude that the elimination of the identified weaknesses did not warrant a rating above satisfactory. Likewise, we have no basis to question the evaluators' conclusion that the addition of a single strength in 1 of 11 areas reviewed did not warrant an increase in MINACT's score. See Tr. at 155. Certainly, MINACT has not established that the correction of weaknesses and the addition of a single strength warranted a perfect score.  (MINACT, Inc., B-400951, March 27, 2009) (pdf)

The protester alleges that the agency improperly focused its evaluation on the final year of the 3 years of past performance that should have been considered, and that the agency has not produced the necessary contemporaneous documentation to support the score [DELETED] given the protester. As a general matter, the evaluation of an offeror’s past performance is a matter within the discretion of the contracting agency, and we will not substitute our judgment for reasonably based past performance ratings. In determining whether a particular evaluation conclusion is reasonable, we examine the record to determine whether the judgment was reasonable and in accord with the evaluation criteria listed in the solicitation. Abt Assocs., Inc., B-237060.2, Feb. 26, 1990, 90-1 CPD para. 223 at 4. Implicit in that examination is that the evaluation must be documented in sufficient detail to show that it was not arbitrary. Federal Acquisition Regulation (FAR) sections 15.305(a), 15.308; Quality Elevator Co. Inc., B-276750, July 23, 1997, 97‑2 CPD para. 28 at 3.

According to the agency, the TET members relied primarily on an oral discussion of their individual experiences with HTS in arriving at the past performance rating of [DELETED]. Supplemental AR at xviii. There is no contemporaneous documentation of that discussion. In an evaluation that takes into account the agency’s own knowledge of offerors’ performance, the fundamental requirement that evaluation judgments be documented in sufficient detail to show that they are reasonable and not arbitrary must still be met. Omega World Travel, Inc., B‑271262.2, July 25, 1996, 96-2 CPD para. 44 at 4.

The agency argues that evidence of HTS's past performance, first introduced into the record as part of the agency report on the protest, supports the reasonableness of the evaluation, citing Omega World Travel, Inc.. In that case, however, the record showed that the agency evaluators relied in their deliberations on specific evaluation material subsequently produced in the protest record. Here, there is no contemporaneous account of the discussion of HTS's past performance, let alone one that references the evaluators' reliance on any of the written documentation the agency supplied in response to the protest. The TET chair asserts that the team based its evaluation of the protester's past performance on "the written evaluations that it had," Declaration of TET Chair at para.15, but fails to identify any specific document or to state whether the evaluators considered any of the contracts listed in the protester's proposal. In contrast, the TET chair goes into considerable detail, in response to the protest, recounting the personal experiences of the TET members that contributed to the rating. Id. at paras. 6‑14. The documents produced in the agency report may or may not be part of the written evaluations that the TET asserts it relied on, but the TET chair's recollection of the evaluation process indicates that greater weight was given to the undocumented team discussion of past performance.

As discussed above, evaluations must be documented in sufficient detail to show that they were not arbitrary. FAR sections 15.305(a), 15.308. Here, without any contemporaneous documentation of that oral evaluation, and with no record, contemporaneous or otherwise, of what contract performance information was considered and how much relevance the information was accorded, we have no basis on which to conclude that the past performance evaluation was reasonable.  (Helicopter Transport Services LLC, B-400295; B-400295.2, September 29, 2008)  (pdf)

In reviewing protests of alleged improper evaluations and source selection decisions, it is not our role to reevaluate submissions; rather, we will examine the record to determine whether the agency’s judgment was reasonable and in accord with the stated evaluation criteria and applicable procurement laws and regulations. However, for our Office to perform a meaningful review, the record must contain adequate documentation showing the bases for the evaluation conclusions and source selection decision. Panacea Consulting, Inc., B-299307.4, B- 299308.4, July 27, 2007, 2007 CPD para. 141 at 3-4. Where an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for the source selection decision. Systems Research & Applications Corp.; Booz Allen Hamilton, Inc., B-299818 et al. Sept. 6, 2007, 2008 CPD para. 28 at 12.

In our view, the record here is inadequately documented to show the reasonableness of GSA’s decision to exclude compensation management from the protesters’ awards. Additionally, GSA’s arguments in this protest appear inconsistent with the contemporaneous record of the TEP conclusions. While the TEP’s conclusions are fairly general, they do not assign the protesters a “high risk,” a “red” rating, or any other rating indicating technical unacceptability for any aspect of compensation management. In addition, there is no record that the contracting officer arrived at a reasoned independent judgment regarding rating, risk, or acceptability of the protesters’ proposals under the RFP criteria at any time after the December 18 TEP report, which assigned a rating of “yellow” and “moderate risk” to these proposals. The TEP report itself is otherwise silent on the basis for implicitly excluding compensation management from the awards.

In attempting to address the lack of any discussion of how the TEP evaluated the protesters’ approach to compensation management, GSA argued that “[s]ince there was no demonstration of the protesters’ ability to perform using GSA supplied data, there was nothing for the TEP to evaluate or discuss with respect to the award of Compensation Management.” Supp. AR at 2-3.

We think GSA’s argument is not supported by the record. In the RFP, and again in the additional instructions provided before the demonstration, GSA sought significant information about how offerors would provide compensation management services. Very simply, while it is undisputed that the protesters did not demonstrate the Ceridian product using GSA’s sample test data, both protesters provided information on their compensation management approach in their written proposals, and demonstrated the functions of their private payroll processing subcontractor using commercial data. In light of this, we fail to see a reasonable basis for GSA’s position that there was nothing for the TEP to evaluate, and, in fact, the contemporaneous summary ratings given in the final TEP report suggest that GSA did not find the protesters’ approach unacceptable. Moreover, we think the argument that these proposals provided nothing to discuss or evaluate is both inaccurate and unfair, and is belied by the record.

In summary, we conclude that GSA’s decision to exclude compensation management services from the protesters’ awards is not reasonably supported by either the contemporaneous record or the agency’s explanations during this protest. Accordingly, we need not reach the specific issues raised by the protesters concerning the evaluation of their proposals and the meaning of specific provisions in the RFP. (Carahsoft Technology Corporation; Allied Technology Group, B-311241; B-311241.2, May 16, 2008) (pdf)

KC-30 Overrun and Breakaway Capability


Boeing also complains that the Air Force did not reasonably assess the capability of Northrop Grumman’s proposed aircraft to refuel all current Air Force fixed-wing tanker-compatible aircraft using current Air Force procedures, as required by a KPP No. 1 threshold under the aerial refueling area of the key system requirements subfactor. See RFP, SRD sect. 3.2.10.1.1.9. Specifically, Boeing notes that current Air Force refueling procedures require that the tanker aircraft be capable of “overrun” and “breakaway” procedures when necessary, which would require the tanker aircraft to fly faster than the receiver aircraft or quickly accelerate during refueling. Boeing’s Second Supplemental Protest at 29. Boeing contends that the Air Force unreasonably determined that Northrop Grumman’s proposed aircraft would meet these requirements.

With regard to the overrun issue, the record shows that Northrop Grumman was twice informed by the Air Force during discussions that the firm’s initially identified maximum operational airspeed of [Deleted] Knots Indicated Air Speed (KIAS) would not be sufficient under current Air Force overrun procedures to achieve required overrun speeds of [Deleted] KIAS for various fighter aircraft, including the [Deleted], or [Deleted] KIAS for the [Deleted]. See AR, Tab 184, EN NPG-MC1-003, at 2; EN NPG-MC1-003a, at 1-2. Ultimately, Northrop Grumman informed the Air Force that a [Deleted] limited the aircraft’s operational speed, but that Northrop Grumman proposed to include a [Deleted] to achieve the necessary overrun speed. See id., Northrop Grumman Response to EN NPG‑MC1‑003a, at 2-7. The Air Force accepted Northrop Grumman’s proposed solution as satisfying this KPP threshold. HT at 628.

Boeing complains that Northrop Grumman’s proposed solution of [Deleted] to achieve overrun speed requires [Deleted], which is not consistent with the Air Force’s current procedures as is required by the KPP. See Boeing’s Second Supplemental Protest at 29-32; Boeing’s Comments at 64. Boeing also argues that the agency did not note that Northrop Grumman qualified its promise to increase its maximum operational airspeed in its EN response. Specifically, Boeing points out that Northrop Grumman stated that, [Deleted], the KC‑30 had a maximum airspeed of [Deleted] KIAS, and not the [Deleted] KIAS evaluated by the Air Force. See AR, Tab 184, Northrop Grumman Response to EN NPG-MC1-003a, at 9.

At the hearing that our Office conducted in this protest, the Air Force produced its SSET mission capability factor team chief to testify regarding the agency’s evaluation of the capability of Northrop Grumman’s aircraft to satisfy this KPP threshold. This witness, in response to direct examination, stated that the SSET found that [Deleted] would allow the KC-30 to achieve the necessary airspeed to perform the required overrun and breakaway procedures. Specifically, he testified that the SSET was convinced that, by [Deleted], the KC-30 could achieve an operational airspeed of [Deleted] KIAS, because Northrop Grumman had informed the agency in its EN response that the commercial A330 aircraft had a maximum “dive velocity” of 365 KIAS and had been flight tested to a dive velocity of [Deleted] KIAS, and that analysis had been done showing that the A330 could even achieve a dive velocity of [Deleted] KIAS. HT at 626-27. The mission capability factor team chief testified that the SSET evaluated Northrop Grumman’s response to indicate that the [Deleted], see HT at 637-38, and that in any event Air Force current procedures did not require the use of the [Deleted] during aerial refueling operations. HT at 638-39.

From this record, we cannot conclude that the Air Force reasonably evaluated the capability of Northrop Grumman’s proposed aircraft to satisfy the KPP threshold requirement to refuel all current Air Force fixed-wing tanker-compatible aircraft using current Air Force procedures. The contemporaneous record, as explained by the hearing testimony, does not establish that the Air Force understood Northrop Grumman’s response in discussions concerning its ability to satisfy the solicitation requirements, nor does it demonstrate that the agency had a reasonable basis upon which to accept Northrop Grumman’s promises of compliance.

First, we agree with Boeing that the SSET erred in concluding that the [Deleted] in tanker refueling operations was not a current Air Force procedure. See HT at 638, 735; Air Force’s Post-Hearing Comments at 19. As noted above, the contemporaneous evaluation record shows that the agency interpreted the solicitation requirement to comply with “current [Air Force] procedures” to mean compliance with the procedures set forth in the agency’s flight manuals for the KC‑135 and KC-10 tanker aircraft, and expressly informed Northrop Grumman during discussions that the flight manuals for the KC-135 and KC-10 established the current Air Force procedures for refueling operations. See AR, Tab 184, EN NPG‑MC1-003a, at 1, wherein the agency stated “[a]erial refueling procedures were contained in T.O. 1-1C-1-3 and 1-1C-1-33 for the KC-135 and KC-10 respectively when the RFP was released.”[62] These manuals show that current Air Force procedures provide that tanker pilots [Deleted] in refueling operations. For example, the KC-135 manual under Section IV, Air Refueling Procedures, warns tanker pilots that they “must be prepared to assume aircraft control [Deleted],” and under Section V, Emergency Air Refueling Procedures, instructs tanker pilots that in a breakaway situation, if a climb is required, they must “[Deleted].” See AR, Tab 289, Flight Manual KC-135 (Tanker) Flight Crew Air Refueling Procedures, Supp. III, T.O. 1-1C-1-3, Jan. 1, 1987, as revised Sept. 1, 2004, at [Deleted]. Similarly, the KC‑10 flight manual provides under Section III, Air Refueling Procedures, that the “[Deleted].” Id., Flight Manual, KC-10A Aircraft, Flight Crew Tanker Air Refueling Procedures, USAF Series, T.O. 1‑1C-1-33, Sept. 1, 2002, as revised Jan. 31, 2005, at [Deleted]. In this regard, Boeing provided the statement of a retired Air Force pilot, who had extensive experience as both a KC-10 and KC-135 tanker pilot and had operated each aircraft as both a tanker and a receiver in refueling missions; this individual stated:

Refueling is more demanding and difficult for both tanker and receiver aircraft if the tanker [Deleted]. For the tanker pilot, [Deleted]. For the receiver pilot, [Deleted]. Due to these realities, existing refueling guidelines dictate that [Deleted] should be used for refueling under normal circumstances. [Citations omitted.] Beginning aerial refueling [Deleted] should it become necessary, violates this policy. As previously noted, [Deleted].

Boeing’s Comments, attach. 14, Declaration of Retired Air Force Pilot, at 3-4. Although the Air Force and Northrop Grumman generally disagree with Boeing’s consultant that the Air Force’s current procedures provide for the [Deleted], neither the agency or intervenor have directed our attention to anything in the KC‑135 or KC‑10 flight manuals or to any other source that would establish that Boeing’s view, which appears to be reasonable on its face, is in error.

We also find unsupported the agency’s conclusion that Northrop Grumman’s proposed solution of [Deleted] did not also involve [Deleted]. In its EN response, Northrop Grumman informed the Air Force that 330 KIAS was the normal design maximum operating velocity of the commercial A330 aircraft, and that “selection of a [maximum operating velocity] drives overall design characteristics of the aircraft, specifically aerodynamic and structural design limits, handling quality definition, and thrust.” See AR, Tab 184, Northrop Grumman Response to EN NPG‑MC1-003a, at 2. Northrop Grumman explained that its [Deleted] limited the aircraft to its maximum operating velocity, but that the firm could [Deleted] to exceed the maximum operating velocity. The awardee then stated “three cases . . . to illustrate the performance of the KC-30 with and without [Deleted].” Id. at 3. The three cases that Northrop Grumman identified and separately described were (1) KC-30 [Deleted]; (2) KC-30 [Deleted]; and (3) KC-30 [Deleted], which indicated that the KC-30 could only meet the overrun requirement under the third case where both the [Deleted]. Id. at 3-6.

The SSET read, as described by the testimony of its mission capability factor team chief, Northrop Grumman’s EN response to describe a “fourth case” (although not identified as such) under the “third case” heading, but located at the end of that section, where, the agency contends, the KC‑30’s [Deleted] but the [Deleted]. See HT at 664. However, we are unable to accept such a reading of Northrop Grumman’s EN response. It ignores the logical structure of Northrop Grumman’s response to the agency, which only identified and described three cases. Moreover, nowhere in its response to the agency’s EN does Northrop Grumman suggest a “fourth case” where the [Deleted]; rather, the only reference to both the [Deleted] in the third case expressly states that the [Deleted] (“Case 3: KC-30 [Deleted]”).[63] See AR, Tab 184, Northrop Grumman Response to EN NPG‑MC1-003a, at 6. In any event, given the uncertainty surrounding the agency’s interpretation of Northrop Grumman’s solution to a matter the agency believed could render the firm’s proposal unacceptable, see HT at 625, 649, this is something the agency should have continued to clarify and resolve during discussions with the firm.

Even apart from the agency’s apparent misreading of Northrop Grumman’s EN response and disregard of the current Air Force procedure to [Deleted], the record does not establish that the agency had a reasonable basis for concluding that Northrop Grumman’s proposed solution would allow its aircraft to obtain the requisite overrun airspeeds to satisfy this KPP threshold. The witness that the Air Force produced to support its arguments on this point testified that the SSET had concluded that the KC-30 had the “inherent capability” of reaching airspeeds greater than [Deleted] KIAS (the aircraft’s certified maximum operational airspeed) based upon the far greater airspeed ([Deleted] KIAS) identified by the firm for its certified dive velocity.[65] See HT at 624-28; Air Force’s Post-Hearing Comments at 17‑18. In this regard, the SSET apparently believed that simply [Deleted] would enable the aircraft to achieve its indicated dive velocity airspeed as its operational airspeed.

Although the SSET mission capability factor team chief repeatedly testified that the dive speed indicated that the aircraft would have the structural ability to fly at the dive speed limitation, see, e.g., HT at 674, he also admitted under cross examination that he did not know what the relationship was between maximum operating airspeed and design dive speed:

Q: What’s your understanding of what the general margin is between maximum operational velocity and dive velocity?

A: I’m not aware.

Q: Was there somebody on your team that was advising you about what the general margin is or difference is between maximum operational velocity and dive velocity?

A: There could have been. We had advisors for handling qualities.

Q: I know you had advisors. I’m asking you, were there any advisors who actually helped you with understanding the difference between dive velocity and maximum operational velocity?

A: They did not help me, no.

Q: Did they help the team?

A: Not that I’m aware of.

HT at 669-70. The SSET mission capability factor team chief’s (and presumably the SSET’s) lack of knowledge concerning the relationship between maximum operating airspeed and design dive airspeed[66] is particularly troubling given the definition of maximum operating limit speed in FAA’s regulations:

The maximum operating limit speed . . . is a speed that may not be deliberately exceeded in any regime of flight (climb, cruise, or descent), unless a higher speed is authorized for flight test or pilot training operations. [The maximum operating limit speed] must be established so that it is not greater than the design cruising speed . . . and so that it is sufficiently below [dive speed and velocity] to make it highly improbable that the latter speeds will be inadvertently exceeded in operations.

14 C.F.R. sect. 25.1505.

In sum, despite having identified, as an issue for the hearing, the capability of Northrop Grumman’s proposed aircraft to satisfy the airspeed requirements of this KPP threshold, we have been presented with no testimony or documented analysis that explains why simply [Deleted] on the KC-30 would ensure that the proposed aircraft would achieve required overrun airspeeds that are in excess of its FAA certified maximum airspeed.[67] Furthermore, neither the Air Force nor Northrop Grumman has directed us to any documentation establishing that the agency analyzed what would be entailed in designing the KC-30 to exceed the certified maximum operational airspeed limit.[68] Given Northrop Grumman’s recognition in its EN response that selection of the maximum operational airspeed limit “drives overall design characteristics of the aircraft, specifically the aerodynamic and structural design limits, handling quality definition, and thrust,” see AR, Tab 184, Northrop Grumman Response to EN NPG-MC1-003a, at 2, it would seem apparent that some design and FAA re-certification efforts could be necessary.

Boeing also complains that the Air Force did not reasonably evaluate the capability of Northrop Grumman’s aircraft to initiate emergency breakaway procedures when refueling the [Deleted]. Current Air Force procedures, as reflected by the KC-135 flight manual, specifies that the tanker will refuel the [Deleted] at an airspeed of [Deleted] KIAS, see AR, Tab 289, Flight Manual KC‑135 (Tanker) Flight Crew Air Refueling Procedures, Supp. III, T.O. 1-1C-1-3, Jan. 1, 1987, as revised Sept. 1, 2004, at [Deleted], and Northrop Grumman’s EN response indicates that the KC‑30’s airspeed is limited to [Deleted] KIAS with the aircraft’s [Deleted]. See AR, Tab 184, Northrop Grumman Response to EN NPG-MC1-003a, at 9. Boeing contends, citing the statement of its former tanker/receiver pilot consultant, that there is insufficient margin between airspeed at which [Deleted] are refueled and the KC-30’s operational airspeed limit during refueling (a [Deleted]-KIAS margin) to allow for emergency breakaway maneuvers. See Boeing’s Comments, attach. 14, Declaration of Retired Air Force Pilot, at 3-4.

As was true with respect to whether the KC-30 can satisfy the current Air Force procedures with respect to overrun airspeed, there is no documentation in the record setting forth an analysis of whether Northrop Grumman’s proposed aircraft has sufficient operational airspeed when refueling the [Deleted] to initiate an emergency breakaway procedure. The agency’s counsel provided a proffer at the hearing that the SSET’s analysis of whether the KC-30 was capable of performing a breakaway maneuver with the [Deleted] was contained in the SSET’s Final Evaluation Summary Report for Northrop Grumman. See HT at 784; see AR, Tab 215, Evaluation Summary Report for Northrop Grumman, at 3. Neither the page referenced by agency counsel or any other part of that document contains any analysis of whether Northrop Grumman’s proposed aircraft can perform a breakaway procedure while refueling the [Deleted]; rather, the page referenced by agency counsel merely states that “[t]he Offeror has substantiated the ability to deliver a KC-X aircraft that meets (minimum requirement) all KPP thresholds associated with aerial refueling,” and provides no reasons or analysis supporting this conclusion. AR, Tab 215, Evaluation Summary Report for Northrop Grumman, at 3.

Although the SSET mission capability factor team chief was examined extensively about the SSET’s consideration of the KC-30’s ability to perform breakaway procedures, he recalled little about the SSET’s discussions in this regard. His testimony does indicate, however, that the SSET accepted that the KC-30’s maximum operational airspeed when refueling ([Deleted]) was [Deleted] KIAS, and that the SSET apparently believed that, to initiate the emergency breakaway procedure, with Northrop Grumman’s proposed aircraft, the tanker would have to start accelerating and [Deleted] simultaneously. See HT at 706. During cross examination, the SSET mission capability factor team chief admitted that he did not know how long it would take [Deleted] Northrop Grumman’s proposed [Deleted] or what the procedure was for [Deleted], nor was he aware of whether this was ever analyzed by the agency in its evaluation. HT at 685-87, 707.

In sum, we conclude that the record does not demonstrate that the agency reasonably determined that Northrop Grumman’s proposed aircraft would be able to refuel all current Air Force fixed-wing tanker-compatible receiver aircraft in accordance with current Air Force procedures as was required by this KPP No. 1 threshold.  (The Boeing Company, B-311344; B-311344.3; B-311344.4; B-311344.6; B-311344.7; B-311344.8; B-311344.10; B-311344.11, June 18, 2008) (pdf)

ATS asserts that the best value determination was flawed because the agency has provided no documents showing that proposals were evaluated by individual evaluators and because the agency has provided no evidence that it conducted a comparative analysis of the proposals to determine which was the best value.  This argument is without merit. First, the lack of documents prepared by individual evaluators does not render an agency’s evaluation unreasonable per se; rather, we consider the record adequate if the consensus documents and source selection decision sufficiently document the agency’s rationale for the evaluation. Joint Mgmt. and Tech. Servs., B-294229, B-294229.2, Sept. 22, 2004, 2004 CPD para. 208 at 3-4; Global Eng’g and Constr., LLC, B-290288.3, B-290288.4, Apr. 3, 2003, 2003 CPD para. 180 at 3 n.3. Here, the evaluation documentation is sufficient because it includes the evaluators’ consensus report, which details the strengths and weaknesses of the proposals that formed the basis for both the agency’s evaluation ratings for each offeror and the selection decision itself. With respect to the absence of a detailed comparative evaluation of the proposals, since the proposals selected for award were both higher technically rated and lower priced than ATS’s proposal, such a comparative evaluation--i.e., a price-technical tradeoff--was not required. MD Helicopters, Inc., Agusta Westland, Inc., B-298503 et al., Oct. 23, 2006, 2006 CPD para. 164 at 49 n.49.  (Alliance Technical Services, Inc., B-311329; B-311329.2, May 30, 2008) (pdf)

As noted above, the solicitation expressly provided that the agency would evaluate the realism of the offerors’ proposed prices, and that such analysis would include consideration of an offeror’s [deleted]. RFP at 86.

Further, the solicitation provided that the agency would also assess proposal risk, including an assessment of whether an offeror’s proposed approach has potential for “disruption of schedule, increased cost, degradation of performance, and the need for increased Government oversight, as well as the likelihood of unsuccessful contract performance.” Id. at 82.
It appears beyond reasonable dispute that, even in the context of a fixed-price contract, an offeror’s proposed approach that [deleted] could create risks of [deleted].

In order for our Office to meaningfully review an agency’s evaluation, the agency must have adequate documentation to support its various judgments. Where an agency fails to create or retain documentation regarding its evaluation assessments, it bears the risk that our Office will be unable to determine whether the agency’s judgments were reasonable. Southwest Marine, Inc.; American Sys. Eng’g Corp., B-265865.3, B-265865.4, Jan. 23, 1996, 96-1 CPD ¶ 56 at 10.

Here, the agency’s evaluation record includes no meaningful documentation addressing the unexplained changes in Boeing’s assumptions between submission of its initial proposal and its subsequent proposal revisions. Specifically, the record contains no documentation regarding any agency consideration of the basis for Boeing’s changed [deletion], how Boeing’s revised [deleted] correspond to the reality of the [deleted], how the revised [deleted] correspond to the agency’s own [deleted] projections, or whether Boeing’s revised [deleted] are likely to create [deleted]. Accordingly, on the record here, we are unable to determine whether the agency reasonably concluded that Boeing’s proposed price is realistic, or whether the agency’s assessment of “low risk” for Boeing’s proposal, under each of the mission capability subfactors, is reasonable in light of Boeing’s revised [deleted]. Since we are unable to determine whether the agency reasonably performed a price realism analysis, or properly considered the potential risk flowing from Boeing’s revised [deleted], we sustain the protest on this basis.  (Pemco Aeroplex, Inc., B-310372, December 27, 2007) (pdf)

In order for us to review an agency’s evaluation of proposals, an agency must have adequate documentation to support its judgment. Northeast MEP Servs., Inc., B-285963.5 et al., Jan. 5, 2001, 2001 CPD para. 28 at 7. While an agency is not required to retain every document or worksheet generated during its evaluation of proposals, the agency’s evaluation must be sufficiently documented to allow review of the merits of a protest. KMS Fusion, Inc., B-242529, May 8, 1991, 91-1 CPD para. 447 at 10. Where an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for its source selection decision. Southwest Marine, Inc.; American Sys. Eng’g Corp., B-265865.3, B-265865.4, Jan. 23, 1996, 96-1 CPD para. 56 at 10.  Here, the agency’s evaluation record contains the individual evaluator comment reports (over 600 pages), as well as various handwritten notes of SSEB members regarding Apptis’s POC demonstration. AR, Tab 9C, SSEB Evaluation Comments of Apptis; Tab 12, SSEB Notes from Apptis POC demonstration. However, none of these contemporaneous documents contains any reference to either of the nonsubstantive deficiencies that the evaluators found in Apptis’s demonstration. Likewise, there are no other documents in the evaluation record that support the SSEB’s conclusions that Apptis’s personnel had repeated difficulty in getting proposed solutions to work correctly the first time they were demonstrated, or that each of the various components of Apptis’s proposed solution required a different technical expert to implement. Quite simply, while the agency described Apptis’s POC as a “problem plagued demonstration,” AR, Tab 14, SSAC Briefing to the SSA, at 23, it kept no records of which Apptis solutions required multiple attempts, how many attempts were required, or any reasons for the multiple attempts. In sum, we cannot tell if the evaluation of this aspect of Apptis’s proposal was reasonable because the agency record lacks adequate documentation to support its findings regarding Apptis’s POC demonstration.  In its report to our Office, the agency acknowledges that, “[t]here is no documentation from the POC that identifies how many times it took [Apptis] to successfully demonstrate any aspect of [its] proposed solutions.” AR, Apr. 5, 2007, at 9. DISA argues, however, that because the POC demonstration was not separately evaluated, there was no reason for the SSEB to document its concerns regarding Apptis’s nonsubstantive deficiencies. The agency’s argument is based on a fundamental misunderstanding of the requirement that agencies provide an adequate basis for their evaluation findings. While the POC demonstration was not itself a separate evaluation factor, it was a significant part of the agency’s overall evaluation of an offeror’s proposal and, as such, the agency was required to maintain records adequate to permit meaningful review. Tiger Enters., Inc., B-293951, July 26, 2004, 2004 CPD para. 141 at 2.  (Apptis, Inc., B-299457; B-299457.2; B-299457.3, May 23, 2007) (pdf)

Here, notwithstanding the agency’s stated understanding as to what its corrective action would entail, the fact is that the record provided to our Office in response to the protest does not contain sufficient information to support the evaluations or source selection decisions. Under solicitation 446, for example, the record includes a description of the agency’s method for scoring the submissions under the understanding of the SOW, relevant experience, capabilities, and approach evaluation factor. The description states that the agency prepared a matrix that included standards for the award of point scores ranging from 0 points (for an unsatisfactory submission) to 10 points (for an excellent submission), and states further that the evaluators populated this matrix with the raw point scores assigned to the submissions under five subfactors (such as the vendor’s understanding of the SOW, and the vendor’s approach to staff retention). Protest, exh. 1. This explanation goes on to state, mathematically, how the agency calculated the ultimate scores for the price and non-price factors. Id. However, the record does not include any explanation of the bases for assigning the raw point scores in the first place; that is, there is no indication of, for example, any evaluated strengths or weaknesses in the proposals that support the assigned scores. The record does include one evaluator’s worksheets prepared in connection with solicitation 446. Agency Report (AR) exh. 12. The worksheets include some limited comments, but the comments do not relate the evaluator’s conclusions about the submissions to the scores assigned. Further, the worksheets reflect the views of only one of the three evaluators who participated in the evaluation and, thus, in no way can serve as a substitute for a narrative explanation of the consensus scores assigned to the submissions. Protest exh. 2. Moreover, the agency states that, since this evaluator’s worksheets were prepared during the original evaluation, they are irrelevant to the current evaluation. Agency Supplemental Submission, June 15, 2007, at 2. The evaluation materials for solicitation 448 are similarly inadequate. As is the case with the other solicitation, the record contains a document describing the scoring matrix, which identifies the standards for assigning point scores of 0 to 10 as well as various subfactors used in the scoring. Protest exh. 3. Also, as with the other solicitation, the record fails to explain what features of the submissions led the agency to score them as it did. As with solicitation 446, the agency submitted some individual scoresheets for solicitation 448. AR exhs. 9, 11. However, all of these scoresheets were prepared by a single evaluator and contain only brief, often cryptic, notations regarding the basis for the scores assigned. In all of the scoresheets for solicitation 448, the portion of the scoresheet relating to the understanding of the SOW, relevant experience, capabilities and approach evaluation factor is entirely blank, containing neither point scores nor narrative explanation. Id. Further, there are several scoring anomalies that are nowhere explained. For example, the agency assigned one of SID’s employees a score of 13 points under a subfactor with a maximum of 10 points available, and also assigned both a score of 5 and a score of 3 (which were added together) under another subfactor, again without any explanation. AR exh. 9, SID Scoresheet, at 4; exh. 11, SID Scoresheet, at 4. The record also contains no source selection decision document for either solicitation prepared in connection with the agency’s corrective action. Point scores cannot be used as a substitute for adequate documentation showing the bases for the evaluation conclusions reached and source selection decisions made. OSI Collection Servs., Inc., supra, at 8. We conclude that the record contains insufficient documentation to explain the basis for the evaluations and the source selection decisions; accordingly, we sustain this aspect of Panacea’s protest. (Panacea Consulting, Inc., B-299307.4; B-299308.4, July 27, 2007) (pdf)

Here, it is clear that the critical criterion for assessing how many additional linguists will be needed to successfully meet the task order 1 fill rate requirements for annual productive hours is the amount of time an offeror’s linguists are expected to be in a non-productive status. Accordingly, it appears the only logical basis for the agency’s conclusion that GLS’s proposal of [deleted] linguists is more likely to meet the fill rate requirements than L-3’s proposal of [deleted] linguists would be a rationally supported conclusion that GLS’s [deleted] linguists are likely to spend [deleted] less time in a non-productive status than are L‑3’s [deleted] linguists. The record, however, is devoid of any meaningful analysis addressing that issue. Indeed, at the GAO hearing, the MET chair testified that the agency gave no consideration to whether the additional personnel proposed by GLS will be sufficient to compensate for its linguists’ scheduled holidays and sick leave. Tr. at 526. Consistent with this testimony the record reflects no agency consideration of whether GLS’s linguists will spend more or less time in a non-productive status, either scheduled or unscheduled, than L-3’s linguists--nor has the agency represented that such analysis was performed. Absent the agency’s consideration of this issue, reasonably documented and rationally supported by credible data, we are unable to conclude that the agency reasonably evaluated L-3’s proposal of [deleted] linguists as being more likely to create a shortfall against the required fill rate than GLS’s proposal of [deleted] linguists. (L-3 Communications Titan Corporation, B-299317; B-299317.2; B-299317.3, March 29, 2007) (pdf)

In order for our Office to perform a meaningful review of an agency’s award determination, the agency is required to have adequate documentation to support its evaluation of proposals and its award decision. Century Envtl. Hygiene, Inc., B-279378, June 5, 1998, 98-1 CPD para. 164 at 4; Biospherics, Inc., B-278508.4 et al., Oct. 6, 1998, 98-2 CPD para. 96 at 4; Arco Mgmt. of Wash., D.C., Inc., B‑248653, Sept. 11, 1992, 92-2 CPD para. 173 at 3. An award decision is not reasonable where there is no documentation or explanation to support the price/technical tradeoff and where the agency makes its award decision based strictly on a mechanical comparison of the offerors’ total point scores. Universal Bldg. Maint., Inc., B-282456, July 15, 1999, 99-2 CPD para. 32 at 4; see also FAR sections 12.602(c), 15.308. The evaluation record here consists of one‑paragraph summaries of the proposals, and charts showing for each evaluation factor the agency’s assignment of raw scores, the calculation of weighted scores, and the total point score for each proposal. The record lacks any documentation reflecting a meaningful comparative analysis of proposals or any explanation of why Danaher’s lower technically rated, lower-priced proposal was selected for award over Midland’s higher technically rated, higher‑priced proposal. The record shows that the agency relied on a mechanical comparison of the total point scores assigned to the Midland and Danaher proposals without any qualitative assessment of the technical differences between these proposals to determine whether Midland’s technical superiority would justify the payment of a price premium. (Midland Supply, Inc., B-298720; B-298720.2, November 29, 2006) (pdf)

We have reviewed the record here and conclude that the evaluation is not adequately supported. We note at the outset that the evaluation record here is brief, comprised only of the initial evaluation scoring sheets prepared by the individual evaluators (for example, the record does not include the consensus source selection evaluation report contemplated by the source selection plan, AR, exh. 103, at 8); a brief summary of advantages and disadvantages observed during the operational demonstrations; and a brief source selection document. (In this regard, the advantages and disadvantages observed during the operational demonstration are identical to the advantages and disadvantages included in the source selection document.) We find that the evaluation judgments are, in many instances, either factually incorrect, internally contradictory, or so cryptic that we are unable to discern either the basis for the evaluators’ concerns or how their concerns related to the solicitation’s evaluation criteria. (Intercon Associates, Inc., B-298282; B-298282.2, August 10, 2006) (pdf)

A consensus rating need not be the same as the rating initially assigned by the individual evaluators; rather, the final evaluation rating may be arrived at after discussions among the evaluators. I.S. Grupe, Inc., B-278839, Mar. 20, 1998, 98-1 CPD ¶ 86 at 5-6. Where, as here, the agency has destroyed individual evaluation materials, its actions are unobjectionable provided that the consensus evaluation materials relied on by the agency support the agency’s judgments regarding the relative merits of the proposals. Global Eng’g and Constr., LLC, B-290288.3, B-290288.4, Apr. 3, 2003, 2003 CPD ¶ 180 at 3 n.3. We find no merit to this aspect of JMTS’s protest. As noted, the record includes the agency’s consensus evaluation materials which, contrary to JMTS’s assertion, provide a significant level of detail about the evaluators’ findings regarding the strengths and weaknesses identified in the proposals. While JMTS devotes a significant portion of its protest to asserting that the evaluators’ conclusions are erroneous primarily as they relate to JMTS’s proposal, the protester’s disagreement with the evaluation conclusions does not demonstrate that they are lacking in detail. (Joint Management & Technology Services, B-294229; B-294229.2, September 22, 2004) (pdf)

The record shows that the Air Force did not evaluate Adelphia’s “record of integrity and business ethics” as part of its past performance evaluation, as was specifically required by the RFP. The Air Force does not assert that it performed such an evaluation, but merely argues that its “very good” rating of Adephia’s past performance was justified, given the information provided and reviewed. The record shows that the Air Force assessed Adelphia’s past performance as “very good” based only upon the survey responses it received from the three contract sources identified by Adelphia in its proposal and the DCMA pre-award survey. See Agency Report, Tab 9, Past Performance Assessment, at 1; Contracting Officer’s Statement at 3. The past performance surveys, however, did not seek or receive any information concerning Adelphia’s record of integrity or ethics. [6] Furthermore, as asserted by the protester, Adelphia performed these three referenced contracts while the indicted members of the Rigas family had significant ownership interest and control in the awardee and its parent companies. Given the specific RFP language, the charges brought by the SEC and the indictment of the Rigas family members should have been (but were not) evaluated as relevant information in the agency’s assessment of Adelphia’s past performance. In short, because the agency’s past performance evaluation was not in accord with the stated RFP criteria, we find the agency’s evaluation of Adelphia’s past performance to be inconsistent with the RFP evaluation scheme and unreasonable. See Beneco Enters., Inc., B- 283512.3, July 10, 2000, 2000 CPD ¶ 176 at 7.  We also find no basis in the record to find reasonable the Air Force’s evaluation of Southwestern Bell’s past performance. As indicated above, the agency’s entire explanation of its evaluation rating of the protester’s past performance was that Southwestern Bell had a dozen contracts with the Air Force’s procuring office, that these contracts were “well documented,” and that the protester’s evaluated price was higher than that of the awardee. Agency Report, Tab 9, Past Performance Assessment, at 2. Despite the agency’s statement that Southwestern Bell’s performance of contracts with its office were “well documented,” no documentation or explanation of that performance has been provided to support the agency’s evaluation rating, which was equal to Adelphia’s rating. In fact, despite the protester’s specific complaint of the paucity of the Air Force’s explanation, the agency has provided no further information of any kind in support of its evaluation assessment. We find this inexplicable, given that the RFP provided for a qualitative assessment of the offerors’ past performance and for an integrated assessment of the merits of the offerors’ respective assessments and their evaluated price to determine the “greatest value.” See RFP at 20. (Southwestern Bell Telephone Company, B-292476, October 1, 2003) (pdf)

On the record discussed above, we find the agency's post-protest reevaluation to lack credibility. As discussed above, the agency has offered no rational support for having increased AWS's past performance rating. Further, we find this portion of the agency's reevaluation particularly troubling in light of the multiple, conflicting numbers that appear in various post-protest documents regarding the adjusted point scores to be awarded to Dismas's and AWS's proposals. Similarly, the agency's summary assertions that Dismas was not prejudiced by the agency's other procurement errors--including the agency's failure to consider all of Dismas's proposal information, and the agency's failure to permit Dismas to respond to adverse past performance information--are substantially without any documented, objective analysis.[18] On this record, we decline to give any material weight to the agency's post-protest activities and we reject the assertion that Dismas was not prejudiced. To the contrary, had a proper evaluation been performed, we believe there is a reasonable possibility that Dismas's proposal could have been rated higher than AWS's under a majority of the non-cost/price evaluation factors, including the most heavily weighted past performance factor. Since the RFP provided that non-cost/price factors would be “significantly more important than cost[/price],” we conclude that Dismas has a substantial chance of receiving the award in the event the agency properly evaluates Dismas's and AWS's proposals. See McDonald-Bradley, B‑270126, Feb. 8, 1996, 96-1 CPD ¶ 54 at 3; see also Statistica, Inc. v. Christopher, 102 F.3d 1577 (Fed. Cir. 1996).  (Dismas Charities, Inc. , B-292091, June 25, 2003) (pdf)

As indicated above, the evaluation documentation regarding past performance is sparse and conclusory. Where an agency fails to document or retain evaluation materials, it bears the risk that there will be inadequate supporting rationale in the record for the evaluation and source selection decision and that we will not conclude that the agency had a reasonable basis for the decision. Kathpal Techs., Inc.; Computer & Hi-Tech Mgmt., Inc., B-283137.3 et al., Dec. 30, 1999, 2000 CPD P: 6 at 12. In addition to the lack of documentation supporting the past performance evaluation, the SSA's stated conclusions in the evaluation documentation in this area were unreasonable or unsupported by the record.  (M&S Farms, Inc., B-290599, September 5, 2002)  (pdf)

While an agency is not required to retain every document or worksheet generated during its evaluation of proposals, the agency's evaluation must be sufficiently documented to allow review of the merits of a protest. KMS Fusion, Inc., B--242529, May 8, 1991, 91-1 CPD para. 447. Where an agency fails to document or retain evaluation materials, it bears the risk that there may not be adequate supporting rationale in the record for us to conclude that the agency had a reasonable basis for the source selection decision. Southwest Marine, Inc.; American Sys. Eng'g Corp., supra, at 10.  (Northeast MEP Services, Inc., B-285963.5; B-285963.7; B-285963.8, January 5, 2001)

Here, we conclude that the contemporaneous record supports the agency's post-protest position that the proposals in fact were equivalent under the technical factor. Specifically, the reference check questionnaires reflected that the protester and the awardee both had relevant landscape experience on projects of size and scope similar to that encompassed by the RFP, and contained uniformly positive narrative statements concerning their actual performance. For example, with regard to timeliness of performance, one of ERG's references stated it was "always prompt [and] performed within the scope of work," and one of PLMS's references stated that it was "always timely--quick response from [named individual]." AR, Tab 6 at 1, 8. With regard to positive and negative performance factors, the same references stated that ERG's "[p]ersonnel are very good listeners, dedicated to their projects, and willing to do and be better in their field of work" and for PLMS's stated that there were "no negatives[;] very pleased with contractor." Id.  The agency's after-the-fact scoring is thus wholly consistent with the contemporaneous record--and thus is entitled to weight in our review--and the methodology used appears reasonable.  (Professional Landscape Management Services, Inc., B-286612, December 22, 2000)

Under the FSS program, agencies are not required to conduct a competition before using their business judgment in determining whether ordering supplies or services from an FSS vendor represents the best value and meets the agency's needs at the lowest overall cost. Federal Acquisition Regulation (FAR) sect. 8.404(a); Amdahl Corp., B-281255, Dec. 28, 1998, 98-2 CPD para. 161 at 3. However, where, as here, an agency conducts a competition, we will review the agency's actions to ensure that the evaluation and source selection were reasonable and consistent with the terms of the solicitation. Computer Prod., Inc., B-284702, May 24, 2000, 2000 CPD para. 95 at 4-5. For such a competition, the agency should contemporaneously document the basis for its determinations regarding its needs and the FSS supply or service that meets those needs at the lowest overall cost in a manner that is adequate to permit meaningful review; however, in appropriate circumstances, our Office will consider post-protest evidence that is a memorialization of the contemporaneous record. Draeger Safety, Inc., B-285366, B-285366.2, Aug. 23, 2000, 2000 CPD para. 139 at 4, 6; Delta Int'l, Inc., B-284364.2, May 11, 2000, 2000 CPD para. 78 at 4.  (Information Spectrum, Inc., B-285811; B-285811.2, October 17, 2000)

Our review of the record confirms that the Navy did not adequately document its evaluation of proposals and that the documentation and further explanation offered during the course of the protest, including the two-part hearing that was conducted by our Office, fail to demonstrate that the evaluation and source selection were reasonable and supported by the facts.  (Future-Tec Management Systems, Inc.; Computer & Hi-Tech, B-283793.5; B-283793.6, March 20, 2000)

The contracting officer's Source Selection/Technical & Cost Evaluation memo provides only the total technical scores and prices for the competitive range proposals and percentage comparisons between TMI's technical score and price and the technical scores and prices of the other competitive range proposals. The contracting officer's memo contains no hint as to the basis for the scoring of the proposals; in fact, there is nothing in the record which indicates that the contracting officer was ever made aware of the individually noted strengths and weaknesses in TMI's or Teltara's proposals. Indeed, the record before us lacks any evidence that the contracting officer did anything more than make percentage comparisons among the competitive range proposal scores and prices in order to determine which offeror should be awarded the contract.  (Teltara Inc., B-280922, December 4, 1998)

Agency conducted competition on an unequal basis where the awardee's initial proposal was substantially in excess of the page limitation stated in the solicitation, and the agency evaluated that proposal for award without specifically advising and providing the other offerors an opportunity to submit proposals without a page limitation.  (Electronic Design, Inc., B-279662.2; B-279662.3; B-279662.4, August 31, 1998)

5. The ATA’s Evaluation of VariQ’s Past Performance Does Not Demonstrate the Consideration of Relevant Factors

Plaintiff next contends that if the [Department of State’s Office of Antiterrorism Assistance] ATA was correct to evaluate VariQ’s proposal under the remaining criteria in the Past Performance factor description, its evaluation lacked a rational basis because VariQ and its subcontractors did not meet those criteria. Specifically, plaintiff contends that neither VariQ nor its subcontractors “managed a multimillion dollar series of training in a global theatre with many simultaneous iterations requiring expanding staffing resources to support deliveries.” Pl.’s Mot. 32. Plaintiff further contends that neither VariQ nor its subcontractors had “experience in the acquisition, configuration and shipping of computer components.” Id. Although defendant does not dispute plaintiff’s contentions in its response to plaintiff’s motion, and there is support for these contentions in the administrative record, see generally AR 758-75, their accuracy is not relevant. VariQ and its subcontractors were not required to demonstrate that they had managed such a training program or acquired, configured, and shipped computer components. Rather, they were required to demonstrate the ability to perform those tasks.

More problematic is the fact that the administrative record lacks any contemporaneous evidence indicating how the ATA determined that VariQ’s proposal satisfied the criteria set forth in the Past Performance factor description. The Technical Evaluation Team’s assignment of an exceptional rating for VariQ’s past performance was based on (1) a vague assertion that VariQ’s “stated past experience” indicated that there was “a high expectation that [VariQ would] be able to perform or exceed the requirements of the statement of work successfully,” (2) its conclusion that VariQ’s “stated experience [was] centric to cyber training,” and (3) its acknowledgment that VariQ had “[e]xcellent reviews from multiple government and public entities.” Id. at 3295. The contracting officer adopted these findings without further analysis. In short, neither the Technical Evaluation Team nor the contracting officer explicitly addressed, for any of the nine past performance examples supplied by VariQ and its subcontractors, whether (1) “the tasks performed [were] of a similar complexity to the work solicited under this solicitation”; (2) the projects reflected “the ability to manage a multi-million dollar series of training in a global theatre”; (3) the projects included “[m]any simultaneous iterations requiring expanded staffing resources to support deliveries”; (4) the projects included “direct support of an international law enforcement training program”; (5) the projects “illustrate[d] experience in the acquisition, configuration and international shipping of various computer related components”; and (6) the projects reflected “existing OEM (Original Equipment Manufacturer) partnerships and the ability to forge necessary manufacturer relationships to ensure compliance with required licensing and End User License Agreements.” Id. at 455. Herein lies the flaw in this procurement.

As noted above, the evaluation and rating of proposals is within the discretion of the procuring agency. When such actions are being challenged in a bid protest, the court must “determine whether the . . . agency provided a coherent and reasonable explanation of its exercise of discretion . . . .” Impresa, 238 F.3d at 1333; accord SEC v. Chenery Corp., 332 U.S. 194, 196- 97 (1947) (“If the administrative action is to be tested by the basis upon which it purports to rest, that basis must be set forth with such clarity as to be understandable. It will not do for a court to be compelled to guess at the theory underlying the agency’s action; nor can a court be expected to chisel that which must be precise from what the agency has left vague and indecisive.”). Moreover, the agency’s action must “evinc[e] rational reasoning and consideration of relevant factors.” Advanced Data Concepts, 216 F.3d at 1058. However, the Technical Evaluation Team’s evaluation of VariQ’s past performance–twice endorsed by the contracting officer–does not meet these standards because it does not address any of the criteria set forth in the Past Performance factor description; in other words, it does not evince a consideration of relevant factors. Indeed, the narrative summary provided by the Technical Evaluation Team in support of its exceptional rating is little more than a copy-and-paste of the solicitation’s definition of “exceptional.” Compare AR 57 (defining “exceptional” as: “Based on the offeror’s recent/relevant performance record, the Government has a high expectation that the offeror will successfully perform or exceed the requirements. It is unlikely that Government intervention will be needed in order to obtain the required services.”), with id. at 3295 (containing the following narrative summary: “Based on the vendors [sic] stated past experience and the PPIRs [reports] presented[,] there is a high expectation that the vendor will be able to perform or exceed the requirements of the statement of work successfully. It is unlikely that Government intervention will be needed in order to obtain the required services.”). Consequently, the court has no basis for determining whether the ATA considered all of the relevant factors when assessing the past performance of VariQ and its subcontractors. Accord AshBritt, Inc. v. United States, 87 Fed.Cl. 344, 370 (“Because the agency has provided no supporting documentation to explain the scores it assigned to [the protester], this Court cannot determine whether the agency’s . . . scoring of [the protester’s] revised proposal took into account the amplified information on past performance and had a rational basis.”), clarified by 87 Fed. Cl. 654 (2009); see also Fla. Power & Light Co. v. Lorion, 470 U.S. 729, 744 (1985) (“If the record before the agency does not support the agency action, if the agency has not considered all relevant factors, or if the reviewing court simply cannot evaluate the challenged agency action on the basis of the record before it, the proper course, except in rare circumstances, is to remand to the agency for additional investigation or explanation.”). In the absence of a coherent and reasonable explanation from the ATA of its evaluation of VariQ’s proposal under the Past Performance factor, the court cannot ascertain whether the ATA’s evaluation had a rational basis.  (FFL Pro, LLC v. U. S., No. 15-1171C, December 18, 2015)  (pdf)

The FDIC’s acquisition regulations require the Panel to “determin[e] price realism and document[] its analysis in either the [Panel] Report or a written memorandum to the [CO].” APM ¶ 3.210(c) (emphasis added). The Administrative Record contains no written memorandum to the CO, and in the fifteen-page Panel Report the documentation of price realism analysis consists of a single paragraph stating only that such analysis was performed, together with a chart showing each offeror’s overall proposed price:

The pricing proposal of each Offeror was initially evaluated with respect to completeness, reasonableness, and realism. The overall analysis of the pricing proposals was conducted by the Senior Contract Specialist, and a summary spreadsheet was provided to the [Panel] to compare the proposed pricing of the Offerors and confirm proposed pricing covered the initial period and all option periods, in accordance with the price schedule. Additionally, the [Panel] reviewed the more detailed price schedules in the proposals to ensure that all mandatory labor categories were addressed as required by the [Solicitation]. The proposed pricing was as follows:

(table deleted)

Therefore, the Panel Report evidences that the Panel conducted a price realism analysis, but it failed to provide that analysis, or even the conclusions thereof, in violation of APM ¶ 3.210(c) and PGI § 3.214(a). APM ¶ 3.210(c) (requiring that the Panel Report document the FDIC’s price realism analysis); PGI § 3.214(a) (same). Without a description of the facts analyzed, and the reasoning that connects the facts with the conclusions, the court is not in a position to determine whether an agency exercised its discretion in a non-arbitrary manner. See Banknote, 365 F.3d at 1357-58 (affirming a trial court determination that best value analysis documentation requirements were met, because “the contracting officer described why he believed that [three offerors] represented the strongest best value selections, while [two others] did not, and explained the relative strengths and weaknesses of the offerors in the evaluation categories underlying the technical rankings”).

It is a well settled principle of administrative law that a federal agency must “articulate a satisfactory explanation for its action including a rational connection between the facts found and the choice made.” State Farm, 463 U.S. at 43 (internal quotation marks omitted). Although the court “may not supply a reasoned basis for the agency’s action that the agency itself has not given,” the court may “uphold a decision of less than ideal clarity if the agency’s path may reasonably be discerned.” Id. In this case, the Government and MMC conducted the price realism analysis in their briefs and asked the court simply to infer that the FDIC made similar use of the data in the Administrative Record. Gov’t Mot. JAR at 21-22 (citing AR Tab 18.2 at 1053 (MMC’s rates); AR Tab 29 at 1425-28 (GSA schedules)); Int. Mot. JAR at 12-17 (analyzing the labor rates in every proposal and the GSA schedules). Argument, however, is not an acceptable substitute for compliance. For example, the Administrative Record is silent about whether, how, and why the FDIC compared various labor categories in the Solicitation with the labor categories in the GSA schedules, and the court notes that, in fact, the Government and MMC do not agree about which categories are equivalent. Compare Int. Mot. JAR at 14, with Gov’t Mot. JAR at 21. The briefs of the Government and MMC include a price realism analysis that compares MMC’s labor rates to those of . . ., one of the contractors whose rates are in the GSA schedules. Gov’t Mot. JAR at 21; Int. Mot. JAR at 14. But the Government and MMC disagree about how seven of the twelve . . . labor categories align with the labor categories in the Solicitation. See AR Tab 29 at 1427 (showing the . . . rates by labor category); Gov’t Mot. JAR at 21; Int. Mot. JAR at 14. Specifically, MMC considers a . . . “Junior Financial Analyst” comparable to the Solicitation’s “Senior Professional,” but the Government considers the Solicitation’s “Professional” labor category a better match. Int. Mot. JAR at 14; Gov’t Mot. JAR at 21. Confronted with differing post hoc analyses of data in the Administrative Record, the court cannot determine whether the FDIC’s analysis matched the Government’s, MMC’s, or neither. In short, the FDIC’s decisionmaking path cannot “reasonably be discerned.” See State Farm, 463 U.S. at 43. The Government’s citations to Acrow and Blue Lake Forest are unavailing, because in those cases the court inferred only that an agency reviewed and used documents in the Administrative Record, whereas in this case, the Government and the Intervenor ask the court to infer the content of the analysis that was conducted based on the existence of certain documents in the Administrative Record. The Administrative Record documents the result of price realism analysis with respect to only one proposal— ’ (AR Tab 31 at 1440 (Selection Recommendation Report describing it as “unrealistic compared to other labor hour price submissions”)), but there is no document in the Administrative Record that supports this conclusion.  The Government and MMC quote the FDIC’s conclusion that MMC’s proposal “offers fair and reasonable competitive labor rates” (AR Tab 31 at 1446), but that quotation does not establish the realism of MMC’s prices, let alone document or analyze their realism. Compare PGI § 3.210(c)(1) (defining price reasonableness as competitiveness in comparison with the prices submitted by other offerors, the prices charged for similar goods or services, the Agency’s price estimate, or the prices required by law), with PGI § 3.210(c)(2) (defining price realism as an assessment of whether the price “reflects a clear understanding of the requirement and is consistent with the offeror’s technical proposal”). For these reasons, the court has determined that the FDIC’s failure to evidence that a price realism analysis was conducted is not rational.

(sections deleted)

For these reasons, it is hereby ordered that:

This procurement is remanded to the Agency “for additional investigation or explanation” regarding price realism analysis with respect to Solicitation No. RECVR-12-R-0088. See Fla. Power & Light Co. v. Lorion, 470 U.S. 729, 744 (1985) (“If the record before the agency does not support the agency action, if the agency has not considered all relevant factors, or if the reviewing court simply cannot evaluate the challenged agency action on the basis of the record before it, the proper course, except in rare circumstances, is to remand to the agency for additional investigation or explanation.”). The FDIC and its officers, agents, servants, employees, and representatives are preliminarily enjoined from proceeding with or awarding contracts for business operations support services for the FDIC Division of Resolutions and Receiverships, pursuant to Solicitation No. RECVR-12-R-0088 or any related procurement, solicitation, task order, or activity. See RCFC 65(a).  (Cohen Financial Services, Inc. v. U. S. and Mir Mitchell & Company, LLP, No. 13-37C, April 4, 2013)  (pdf)

iv. The Court's Resolution.

It is well established that “procurement officials are entitled to broad discretion in the evaluation of bids and in the application of procurement regulations, particularly in those circumstances where, as here, a negotiated procurement is at issue.” Day & Zimmerman Servs. v. United States, 38 Fed. Cl. 591, 597 (1997) ); see also Oceaneering Int’l, Inc., 2001 WL 695072, at *10 (Comp. Gen. 2001) (“There is nothing improper with the agency identifying strengths and weaknesses under an adjectival rating scheme, as the agency did here. Adjectival ratings and point scores are only a guide to assist agencies in evaluating proposals; information on advantages and disadvantages of proposals is the type of information that source selection authorities should have in addition to ratings and point scores to enable them to determine whether and to what extent meaningful differences exist between proposals. Proposals with the same adjectival ratings are not necessarily of equal quality and the agency may properly consider specific advantages that make one proposal of higher quality than another.”) (citations omitted). (p. 34 of decision in pdf.)

(Sections deleted)

Again, this new addition to CIS’s Final Proposal was considered a “significant strength.” Id.19 The Administrative Record, however, contains no discussion nor analysis of how this rating affected CIS’s “Technical Approach” Subfactor point score and any score increase. Id.; see also TR at 62-66. More importantly, the SEB improperly attributed “[deleted]” benefits of the new “continuous improvement program” element to the “Technical Approach” Subfactor, contrary to the Solicitation, that includes only four subfactors, [deleted]. See AR at 2610. Moreover, as the Government touted, CIS’s proposed [deleted] was particularly influential in the SEB’s increased rating and point score increase for the “Technical Approach” Subfactor, as it was estimated to be “[deleted] over WSI.” See Gov’t Resp. at 17. The FAR, however, considers such “[deleted]” savings to be a matter of “price.” See AR at 26360 (citing FAR Clause 1852.216-83). (p. 40 of decision in pdf.)

The SEB’s point scores are entitled to deference, but only if the underlying decisions properly are explained in the Administrative Record. See, e.g., Femme Comp, Inc. v United States, 83 Fed. Cl. 704, 768 (2008) (need for adequate agency documentation); 210 Earll, L.L.C. v. United States, 77 Fed. Cl. 710, 720 (2006) (holding that an agency is required “to provide ‘a coherent and reasonable explanation of its exercise of discretion[.]’”) (citation omitted); Opti-Lite Optical, 99-1 C.P.D. ¶ 61, 1999 WL 152145, at *3 (1999) (“While adjectival ratings and point scores are useful as guides to decision-making, they generally are not controlling, but rather must be supported by documentation of the relative differences between the proposals, their strengths, weaknesses and risks, and the basis and reasons for the . . . decision.”) (emphasis added); see also Ralph C. Nash & John Cibinic, “Source Selection: A Variety Of Agency Guidance,” 3 No. 8 Nash & Cibinic Rep. ¶ 60 (August, 1989) at 4 (“There is a slow trend toward conferring a significant amount of discretion on source selection officials. This is exhibited by the number of documents prohibiting numerical scoring of certain factors and the requirement in most of them that evaluators prepare substantial narrative justification for the scores the give.”) (emphasis added).

In this case, the court has determined that the SEB violated the APA by failing to create a record to explain and justify the [deleted] increase in point score, or [deleted]% increase, between the SEB’s Preliminary and Final Findings as to CIS’s “Technical Approach” Subfactor, so that the court can determine whether the SEB acted in an arbitrary and capricious manner. See Motor Vehicle Mfrs. Ass’n v. State Farm Mut. Auto. Ins. Co., 463 U.S. 29, 43 (1983) (“[T]he agency must examine the relevant data and articulate a satisfactory explanation for its action and the choice made . . . In reviewing that explanation, we must consider whether the decision . . . was based on a consideration of the relevant factors and whether there has been a clear error of judgment.”) (emphasis added).  (Wackenhut Services, Inc., v. U. S. and Coastal International Security, No. 08-660C, December 15, 2008) (pdf)

Cubic offers citations to a number of Government Accounting Office decisions on the requirement to document analyses. The Court notes this authority and concurs in it. See e.g., S-Cubed, B-242871, 91-1 CPD ¶ 571 (June 17, 1991)(implicit in requirement that agency judgment be reasonable is requirement that judgment be documented in sufficient detail to show it is not arbitrary). But the real question is how much documentation is necessary to trace the decision-makers analysis. More or less analysis may be required depending upon the circumstances.  (Cubic Defense Systems, Inc. v. U.S. and Metric Systems Corp., No. 99-144C, December 3, 1999)