In the January issue of “Contract Management” magazine there is a “Speaking Out” article titled, “Contract Litigation—Where’s the Proof?” written by a Ms. Hettenhouser.

Per Ms. Hettenhouser, “The GAO or the various boards will consider only the record (consisting of transcripts and contemporaneous evidence) in making their rulings and recommendations.”

I did a quick word search using “documentation” and “contemporaneous.”

Future-Tec Management Systems, Inc.; Computer & Hi-Tech Management, Inc., No. B-283793.5; B-283793.6, March 20, 2000

“That is not to say that our Office, in determining the reasonableness of an agency’s evaluation and award decision, limits its review to the contemporaneous evaluation and source selection documentation. Rather, we will consider, in addition to the contemporaneous documentation, information provided to our Office for consideration during the protest, including the parties’ arguments and explanations and testimony elicited at a hearing. Southwest Marine, Inc.; American Sys. Eng’g Corp., supra. However, in considering the entire record, we accord greater weight to contemporaneous evaluation and source selection material rather than judgments made in response to protest contentions. Boeing Sikorsky Aircraft Support, B-277263.2, B-277263.3, Sept. 29, 1997, 97-2 CPD 91 at 15. Where post-protest explanations simply fill in previously unrecorded details of contemporaneous conclusions, we will generally consider them in our review of the rationality of selection decisions, so long as those explanations are credible and consistent with the contemporaneous record. NWT Inc.; PharmaChem Labs., Inc., B-280988, B-280988.2, Dec. 17, 1998, 98-2 CPD 158 at 16.”

One consistent theme in Acquisition Reform is the exhortation to be more like a private business and less “risk averse.” On the other hand, the 1102 is still given broad brush advice to “document the file” to prevent numerous potential ills.

Just for clarity: I agree with Ms. Hettenhouser that it is important to document the file. On the other hand, I think that when you look at the cases you will find that in the cases where the government lost, the documentation was either nonexistent or incoherent.

For discussion:

Are we documenting the file more or less?

Are we eliminating redundant or low value documentation?

Doesn’t it require a high quality of professional judgement to know what needs to be documented and how much?

Why kind of documentation should we be doing in this new reformed and digitized world?

Eric

By susan marie paolini on Wednesday, January 03, 2001 - 09:07 am:

Eric,

I zeroed in on Ms. Hettenhouser's article, having just gone through very instructive protest(s).

Wish I had read her article a year ago - and gotten legal briefing on impact and relevance of various documents and approach (e.g., handling e:mail documents/developing logical ways to file electronic documents so production for discovery requests is less intensive)) as well as practical application of attorney client privilege/work product.

I think the million dollar question is your last one, "What kind of documentation should we be doing in this new reformed and digitized world?" -can anyone recommend reading on this one?

By Stan Livingstone on Wednesday, January 03, 2001 - 10:50 am:

Eric,

My though is do enough documentation to support the reasonableness of decisions made - no more, no less.

Several years ago, I was involved in a complicated protest at the GSBCA. We almost lost because the CO insisted that every piece of paper generated, including hhand written notes of evaluators, be put in the file. One such note gave the appearance of inconsistency with the decisison. It took a lot of talking to save the award.

We all seem to go overboard with documentation when it's not necessary. The test is whether someone removed from the transaction can look at what was done and see if it makes good sense.

Stan

By Kennedy How on Wednesday, January 03, 2001 - 12:37 pm:

Stan,

Some of our COs are the same way when it comes to documentation, but in my limited experience, we didn't require hand-written notes be put into a contract file. That didn't mean that I kept my own documentation, in some cases relatively comprehensive, on occasion if the situation warranted it. Mainly, this was to refresh my own memory somewhere down the line if I needed to follow up on something, or if some issue was going to go on for some length of time.

These documents weren't a part of the contract record; there's the standard memos and correspondence for that, but I've found that having a side "diary" was helpful when it came to litigation. I remember my "misc" folder was at least 3" thick.

Back to the original question, certainly, there needs to be documentation to back up any decision of this type. The more complete it is, the better to bolster your case. I remember being introduced to the phrase "arbitrary and capricious" early on in my career, and the way to avoid this was complete documentation.

I don't think handwritten minutia need be included, but if you're team has disagreeing thoughts, those should be addressed in the final evaluation. I thought we covered something like this in another thread about source selection teams and their education requirements?

Kennedy

By Ramon Jackson on Wednesday, January 03, 2001 - 07:27 pm:

With regard to electronic media the disturbing trend to investigate or pull into court e-mail archives should be considered. We've long recognized that free flow of views, sometimes even short sighted, erroneous or even ill advised views are necessary for sound decisions. The working papers that are a part of that "sausage making" have been regarded as having some protection under the concept that threat of hindsight review will squelch open exchanges to the detriment of decisions.

Studies have shown people seem to regard e-mail as less permanent than a paper note. We read about ill advised business comments spread by e-mail in addition to foolish personal notes in a business setting.

Now it appears e-mail is less subject to disposal than those personal notes that end in landfill, recycling or shredders. An e-mail comment one might regret years later doesn't get wadded up and thrown in the trash. It passes through "systems" whose mass backups may live almost forever. Courts have been ordering them produced as evidence. For example, in the Microsoft case one might wonder if some of the damaging comments would have made it to court if the originator had jotted it on a memo pad and left it on the recipients desk.

Law and custom in this area are evolving and not entirely clear. Anyone using electronic communications or records in sensitive acquisition matters should consider this oddity.

By Kennedy How on Thursday, January 04, 2001 - 11:52 am:

Ramon,

That's an interesting point. I know that during some of my litigation involvement, we had e-mails between myself and the functionals going back and forth over various issues. Not all of them made it into the "official contract record". Obviously, the ones that did were the authorizations, requests for actions, or disposition of some contractor request. These were printed and made a part of the record.

On the other hand, those "I can't see how anybody could request such a thing. What are they thinking? Do they really think this is a good thing?" type mails were treated as "discussions" on the situation. Such as would be found in internal staff meetings. Something like that may well be detrimental to somebody's position, but it didn't become the "official" decision by the Government; indeed, it's just a hashing out of all the options. Certainly, it's our job to review all the options, pros and cons, so that an informed decision can be made.

But, if e-mail records are permanent, and can be produced as "evidence".......

Kennedy

By joel hoffman on Thursday, January 04, 2001 - 02:02 pm:

My personal opinion is that most pre-decisional information, such as individual notes, e-mails, individual evaluation sheets, etc. are just that - "pre-decisional".

If your source selection plan and instructions are correctly written, they should clearly indicate that the evaluation team will perform a "consensus evaluation" and will document that process.

I personally don't provide any place on the evaluation sheets which individuals use during their own proposal review for a "score" or "rating". After the consensus meeting, I collect up the individual notes and throw them away.

The data, relevant to a source selection recommendation made to the selection offical, are the solicitation provisions, initial and revised proposals, source selection plans, instructions to evaluators, the documented consensus technical evaluation narratives, cost evaluation reports, consensus ratings, records of discussions and other correspondence with offerors, cost-technical trade-off reports, the evaluation team report(s), describing what happened and the results, plus report(s) and other official input provided to the team from support team(s).

Of course, the source selection offical(s)' official reports and decision memoranda are also relevant.

If properly documented, anyone should be able to understand how the process worked, the results and full justification for the decisions made and the actions taken.

Individual team member's initial thoughts, informal discussions and notes aren't germane to the selection recommendation or decision. Happy Sails! Joel

By Ramon Jackson on Thursday, January 04, 2001 - 03:25 pm:

I'd have to do more research, but I have the impression litigation has changed things. So far it may be outside the contract world, but it is lurking. I'm sure both Microsoft and the White House considered e-mails just informal exchanges that were background and never to see the light of a courtroom or investigation. Both were wrong.

We tend to lose common sense when it comes to digital things. I used to have real trouble reminding people that normal information security also applied to information that was "01" format. The fact it was digital did not erase all the rules and require complete rethinking. Sure, the issues of emissions, compact size (much easier to walk out with a 3.5 than equivalent paper) and ability to move things rapidly and invisibly added complications, but computers did not require complete reinvention of information security.

Reporters have thought to go through trash in alleys, but plaintiffs or courts would not tend to think of demanding the handwritten notes taken during a process of a year ago. Nobody was going to try to find it in the dump and if recycled or burned it was really gone. Now they have discovered the little mother lode of system backups with all that chit chat that did not die. Reliance on a cloak of "informal discussion" may be no cloak at all.

If I were an attorney representing a company convinced the fix was in and the record had been polished during an acquisition relying on electronic files I'd sure make a run at the backups. The discovery could be absolutely wonderful for my case. Just the damage and damage control dealt the other side could be a major boost: Wow! Got the agency head's ill advised little note to several key acquisition participants urging a tilt toward the winner. Gotcha! Let's hear again under oath just exactly how that e-mail made no difference.

Unless there is a firm shield in law protecting all written material prior to a certain point or of a certain nature we may find the old "protection" was mainly one of practical recovery and not much else. Perhaps some of our legal lurkers might chime in on this.

By joel hoffman on Thursday, January 04, 2001 - 04:12 pm:

Ramon, A recent source selection, which I chaired, was protested to GAO by the incumbent, an unsuccessful offeror. The Protestor asked for any and all personal notes, individual evaluation sheets, etc. We successfully argued that they were not germane to the issue.

This was reassuring, because all notes were made on the evaluators' individual sheets provided each evaluator for writing notes and preliminary narrative comments on. They were discarded after the SSB developed their consensus evaluation.

The source selection board, advisors and KO followed the evaluation plan and instructions, as described in the source selection plan, consistent with the RFP. The official documentation, including backup past performance interview documentation, served as an adequate record of what occurred during the price/technical evaluation, trade-off process and selection decision. There was no need or intention to individually evaluate or score the proposals.

I don't know what happens elsewhere, only my personal experiences. Happy Sails! Joel

By Anonymous on Friday, January 05, 2001 - 07:54 am:

Very recent personal experience in GAO protest includes discovery requests for all e:mails, drafts, and material on computers. My personal opinion is that all this material will be requested during litigation and you may be required to produce it. In any event, with the common practice of forwarding e:mails to alcon, expect others (perhaps even your protestor) to have copies of things you may not have even seen.

By joel hoffman on Friday, January 05, 2001 - 09:06 am:

I wouldn't disagree with you, Anon. E-mail doesn't disappear. I meant to emphasize that I don't think there is a legal requirement to retain personal notes of the evaluators, when the source selection plan states that the board will use consensus evaluations and that minority opinions will be included in the reports for the selection official's consideration. So I throw the notes away before sending up the source selection documentation. Somebody can't get what doesn't exist and since it is not germane to the offical selection decision, there is nothing wrong with not preserving notes. Does anyone retain drafts of official letters, mods, prenegotiation objectives, PNM's, etc. for other actions which could end up being litigated some day? I don't think so - they only keep the official file records. Happy Sails! Joel

By Eric Ottinger on Friday, January 05, 2001 - 10:13 am:

All,

I think Joel has it about right. I would only add that there is plenty of Comp. Gen. precedent to back him up.

The Comp. Gen. wants to see what the SSA was looking at when he/she made the selection. Generally, the Comp. Gen. doesn’t want to go any lower than that, unless there is something really grotesque. In one instance the workbooks were empty. In another instance the workbooks were filled out before the team opened the proposals. Short of this kind of absurdity, the Comp. Gen. makes a point of not being interested in the workbooks.

It’s a shade of difference. I wouldn’t destroy any piece of paper specifically because I anticipate litigation. And I wouldn’t destroy anything after litigation starts. However, similar to Joel’s approach, I would insist that all comments go into the workbooks. Any other pieces of paper should be destroyed as a matter of routine.

If the workbooks reflect significant differences of opinion, there should be some trackability to show how the issue was resolved. Differences of opinion are good. It shows that you take things seriously. Differences of opinion only bite you when you don’t address or resolve them in an appropriate manner.

As I’ve said earlier, I’m not sure that I know what “consensus” means. (It might be that the Chairman just decides what the “consensus” is going to be. Or it may be that the Chairman appoints a subcommittee to decide what the consensus is going to be. Such things have been done without particularly upsetting the Comp. Gen.)

Generally, I take the view that the process of working things out within the team and moving toward consensus is much more important than the initial ratings and comments.

If we say that we want complete consistency from the initial comments to the final briefing, the real message is that we don’t want the evaluators to approach the problem with an open mind or to consider more than one “approved” point of view. I wouldn’t send that message.

Regarding the e-mail. Heck yes, it might be cited in court. Take a look at what happened to Bill Gates. If you aren’t confident that you know what you should put in writing and what you shouldn’t, and when to shut the door to the office, you are in the wrong line of work. Knowing how to be very open and above board and not disclose matters which really shouldn’t be disclosed is part of the job description.

My advice would be to be alert but not to be intimidated. The opposing lawyers are going to do their best to embarrass us, even if they can’t win the case. Our lawyers don’t like to be embarrassed, even if we win the case or protest.

If we did business with no purpose other than making our lawyers happy, we would function at all times in a defensive crouch.

The benefits of e-mail are much greater than the liabilities.

Eric

By Ramon Jackson on Friday, January 05, 2001 - 01:48 pm:

I think several here are somewhat missing the point. Part of the protection afforded informal give and take has been the trash can and either a very messy or destructive fate. One reason these items may not have appeared in past cases may well have been the simple fact that fate made them moot. I think Joel's argument in particular falls into this trap: "Does anyone retain drafts of official letters, mods, prenegotiation objectives, PNM's, etc. for other actions which could end up being litigated some day? I don't think so - they only keep the official file records"

Think again Joel. If those drafts and such are done on a system they are very likely "kept" for varying lengths of time. despite what the drafter thinks. There are ways to insure they are not so kept. It requires pretty much the same system approaches used in classified systems.

Instituting scrubbing after the fact could lead to serious legal consequences solely on that basis. I'd have to check references, but I believe there is already precedent that destruction in accord with a pre-existing policy is legal. In fact, nobody really questions your throwing hard copy notes and worksheets into the trash. They are accepted as gone and demands for them mitigated by the fact recovery is either extremely dirty and unlikely or impossible. Not so with system back ups, hard drives, and other non-scrubbed media.

My point is simply this: Recognize digital information may have a life you are not aware of or even recognize. If your claims of it being "not germane to the issue," internal working papers and such are breached the digits may remind you of a vampire arising from the tomb.

If you are using digital systems in sensitive procurement areas it may be wise to have some expert understanding of how the digits are moved about, archived, and whether or not they are scrubbed (as in security systems). It is probably wise to have a digital destruction policy in existance before you are surprised to find they really are not like little scraps of paper.

By joel hoffman on Friday, January 05, 2001 - 06:18 pm:

Good Grief. I can't wait until I can retire to sanity.

I need to start shaking in my boots that a computer sleuth will be around to reconstruct deleted files - oh, there must be some frigging regulation against writing over draft word documents or Excel spreadheets as I save an update - or perhaps a law against defragging my hard drive after deleting excess draft or temporary files.

By the way, I couldn't care less if someone wants to drag E-mails out of the backup logs because I don't conduct evaluations by E-mail.

Some of you folks must have had some terrible experiences in source selections. I've chaired or been a member of about 90 of them, with little trouble. I've never had any problem reaching consensus of the team - any time anyone is uncomfortable, we document their opinion for the source selection official. I've never been unduly influenced by an outsider, so don't have that perspective, either.

We don't have anything to hide but don't keep individual notes because we don't perform "individual" ratings. If every piece of note paper and every initial comment was compelling, it would prove you only need one evaluator. The reason there are numbers of eyes and brains on a panel is to ensure you get the full picture - hence the consensus method. Seldom if ever would each and every "initial" noted comment represent the final opinion of that evaluator or of the group.

Notes are only cannon fodder for lawyers to skew and twist facts. As I said before, there is no need to keep unofficial notes to clog up the files or to confuse somebody later on. If you properly write up your evaluations, records of discussions, trade-offs, and reports, you have all the necessary documentation to support the decision and how you got there.

I don't think a jury has to keep a complete record of its deliberations for even a death penalty decision. Watch "Twelve Angry Men" sometime. It was 11-1 for guilty in the beginning and ended up 12-0 for acquittal. They handed the judge one slip of paper. Why do some people want to overcomplicate everything they do, because they work for the Government? In the end, it really ain't that important, compared to life and death jury trials!

Yep, I've been protested, 3 times and prevailed in the end 3 times. Happy Sails! Joel

By Ramon Jackson on Friday, January 05, 2001 - 07:09 pm:

Joel, "writing over draft word documents or Excel spreadheets as I save an update" deletes nothing. Depending on internal settings it may in fact simply make the previous copy a backup file. Even if set to make no backup, the old file is quite probably still on the disk and clearly readable without a huge amount of technical resources for a considerable time. As time goes by the deallocated space may be overwritten, but as disk drives have become relatively huge that probability lessens.

Secure systems have software that overwrites the physical disk areas multiple times with meaningless combinations. That is the only way you actually delete a file on a hard drive. Rumors have it that really sophisticated specialized organizations can even recover bits of that. That is why physical destruction of media was required.

While you may be paper based and relatively safe, many agencies were doing or migrating toward activities like source selection using collaborative electronic tools for the non-record process. One very large DoD acquisition years ago had the goal of 100% digital source selection. Commercially available tools are being used. One (I won't mention names since that became the other plague on the old WC) was being pushed in several agencies to my knowledge.

People doing so should be aware of the risk. Many are not. Many treat these informal uses of electronic media as if they were telephone calls or chats in a room. They are not, unless you are used to having bugged phones and rooms.


This quote from Managing the Risks of Corporate E-Mail illustrates: "Increasingly, such forgotten e-mails are becoming the "smoking guns" in lawsuits. . . . Many e-mail users fail to appreciate that e-mails have a lingering existence, and so they tend to treat the medium with far less care and judgment than they would give to traditional correspondence. Users often think of e-mail as if it were a phone call, which is informal, private, and transitory. Many think nothing of using e-mail to pass along a sexual remark, an off-color joke, a derogatory comment, or even sensitive business information. However, a phone call ends with the hang-up; a remark in an e-mail lives on in a record just as if it were a letter or a memo to the file." Litigators know this and seem to be treating it as a new resource.

While we've mainly mentioned e-mail, it is pretty much applicable to all computer based communications and even your wordprocessing, spreadsheets, etc. on a PC. I'd have thought this was clearly known now, but apparently many still have a blind faith that because they can't find the old file it is gone.

Many commercial firms and law enforcement agencies do recovery routinely. If you ever have the misfortune to find a judge ordering examination of electronic media you will probably be surprised at what "turns up."

By joel hoffman on Saturday, January 06, 2001 - 10:31 am:

Thanks for the informative warning, Ramon. Happy Sails! Joel

By Ramon Jackson on Saturday, January 06, 2001 - 12:47 pm:

Joel, the recovery of "deleted" files used to be very easy as both CPM and DOS had commands to do this. CPM even had a capability to search the disk for words, including "deleted" documents. Both Mac and Windows dropped the capability within the normal user environment, but you can buy third party software that will find and undelete files easily.

Deleted computer files can be damning mentions some software as well as providing a pretty fair coverage of the technical issues. One thing people here might consider is commercial third party software that scrubs disk sectors upon delete. It may not be up to national security standards, but it is probably good enough. Your systems and security staff should have the answers and be able to recommend solutions. In my opinion any agency not implementing these precautions is in violation of basic security measures for sensitive, unclassified data.

As an aside, I was helping set up old computers for a school that had been obtained from an agency in the D.C. area. I was surprised and both amused and {not} amused to find the hard disks full of internal documents and memos. Some were quite revealing of internal politics and even had home phone numbers of people to contact in emergency. I did some specific looking for acquisition things and did find some "strategy" papers. For grade school use we considered HD reformat good enough to protect an agency unwilling to protect itself. "Do you know where your old computer is today?" might be an interesting question.

It seems to me that a rational approach to cut off this litigation trend is to restore the paper trash environment to the digital world. Law here is evolving and some judges do really weird things. One Federal Court even authorized e-mail subpoenas, something that to me only illustrates the absolute lack of knowledge of how things really work and lack of judgment.

Technically we can make the bright idea to search digital "trash" as unrealistic as searching physical trash. Even better would be legislation making clear the line between a public record and internal working document for these purposes. Then policy would probably be better implemented applying the technical solution to truly trash the digital working documents.

Some interesting references:
Rand Issue Paper, see particularly "The legal status of electronic transactions" as it has some implications for electronic contracting.

An issue that is cropping up in connection with state public record laws [I've run into a number of state sites with the issue.] is well defined at Don't hit that "DELETE" button while pretty much ignoring the latency of e-mail as we have been mentioning. It does cover the issue of a policy. Another at the same place, Who is the "Custodian" of stored e-mail? could also apply to your files if you are on a network with automatic backup. If "your" PC is on the network and the administrator has configured automatic backups . . .?

As for that judge, see Dodge this: e-mail subpoenas.

By Kennedy How on Monday, January 08, 2001 - 12:28 pm:

Joel,

I'm with you in that I have nothing to hide, but that doesn't stop some clever litigation team from digging around, pulling up whatever innocuous stuff they find, and making a "federal case (pun intended)" out of it. Ramon put it best when he cited electronic media as another resource. It's there for whoever to make whatever out of it.

Kennedy

By Ramon Jackson on Monday, January 08, 2001 - 12:52 pm:

Kennedy, a real concern elsewhere seems to be the "gotcha" aspect. We can probably all remember situations in which some slightly smelly comment or memo was involved that when taken in context can be shown to have been simply ill advised and not an undue influence. Litigation, or at least the PR aspects, now seems to be all too reliant to use the attention getting, suspicion raising comment to at least cause serious disruption to the other side.

Two paragraphs from "Deleted computer files can be damning" (reference above) illustrate:

Under normal operations, Microsoft's Windows and popular applications such as Word and Excel can scatter temporary files, backup files and swap files all over your hard drive, often in fragments.

"There's pieces of deleted files that are scattered across the disk drive -- they could be one or two sentences or two paragraphs, but it's usually enough to sink somebody's boat in a court case," said Joan Feldman, president of Seattle-Based Computer Forensics Inc.

There is some controversy between Archives and agencies about digital, particularly e-mail, record retention responsibility. My impression, possibly out of date, is that a clear policy on permanent record vice working files implemented before any investigative demand for data, honoring legal requirements for public record keeping and followed by consistent application of "working paper" deletions tends to be both wise policy and pass legal muster. It seems a reasonable way to put a stop to fishing in this somewhat new pond.

By Kennedy How on Tuesday, January 09, 2001 - 12:43 pm:

Ramon, this is probably straying a bit from what we'd expect to find in our arena, but during criminal investigations, the first thing they seize is the PCs. Then, you Computer Forensics guy does his thing. The problem that he espouses is that one or two sentences, taken out of context, and absent any other evidence that would mitigate said sentences, will stand alone; for better or worse (in our discussion here, for worse).

Getting back to the e-mail/electronic record, I can see while the "official" evaluation/decision process is non-electronic, we can't say the same for "informal discussions" between members, or even for research purposes. Unless the source selection puts down a "sequestered" decree, there's really no way to stop it.

While your last comment makes sense, the thing about archived e-mail is that it's a kept record, irregardless of what the policy is on "working discussions".

Kennedy

By Ramon Jackson on Tuesday, January 09, 2001 - 03:08 pm:

Kennedy, we have gone way overboard in being a "gotcha" society and that includes some court cases I've read about. It seems to coincide with the lazy brain "zero tolerance" policies such as those that have caused kids to be expelled for bringing a plastic knife to school to spread the peanut butter -- thus potentially commiting two horrendous violations threatening the New American Way (NAW?) and life-as-we-now-know-it.

My recommendation is that any agency using IT/digital in procurement sensitive work -- and here I'd extend that to litigation sensitivity in this sense -- get in touch with their IT security staff and review the issues. From that a written policy on maintenance and actual deletion of electronic material can be followed by implementation.

If an agency is moving toward all digital selections, as some appear to be doing, it is an important consideration. Oddly, people get all worked up about encryption in transit and such without a thought of what kind of collection may be taking place internally as part of "systems" operations.

We don't save sticky notes, record our telephone and other informal conversations, or even keep every draft of things intended for the record as a potential gold mine for litigants. There is no requirement or need to do the equivalent on our systems from ignorance and being careless.

At least one agency, according to a conversation this week-end, has a policy of deleting transitory e-mail without back-up. They have the policy established so that record items are identified and saved, the rest goes to digital death in 30 days.