Declaratory Judgment Approach to Burdensome Pre-Litigation Preservation Demand Tried and Rejected: Is this a Case of “No Good Deed Goes Unpunished”?

Guest Blog: by Michael Simon

In the final paragraph of last week’s post Ralph made the following suggestion on a potential way to deal with an onerous pre-litigation hold demand:

If all else fails, and the potential dollar exposure justifies the expense, a preemptive suit for a declaratory judgment may even be appropriate. You will be breaking new ground to be sure, but as the landscape of litigation changes, new strategies such as this should be considered to cope with the new challenges these changes present.

Unbeknownst to Ralph, at the very time he was writing those prophetic words, the United States District Court for the Eastern District of Texas was giving a thumbs down to that very idea. The case is Texas v. City of Frisco, 2008 WL 828055 (E.D. Tex. Mar. 27, 2008). In reality, as I will explain below, this may be a case where two rights make a wrong: The plaintiff did such a good job of complying with the litigation hold that the Court could not understand plaintiff’s problem.

In Frisco, the State of Texas was faced with the following dilemma: The Texas Department of Transportation (”TxDot“) was planning to convert portions of State Highway 121 from a freeway to a toll road. Apparently a portion to be converted went through the City of Frisco. The City seemingly did not want the pleasure of paying tolls and was (and perhaps still is) considering filing a challenge to the Environmental Re-Evaluation of the State Highway pursuant to the National Environmental Policy Act (”NEPA”), 42 U.S.C. §§4331-4347. As a kick-off to this potential litigation, on April 13, 2007 the City sent TxDot a letter entitled Notice Regarding Preservation of Electronic Data.

This preservation demand does (perhaps intentionally) a poor job of specifying the subject matter of the demand. It broadly states that TxDot should preserve “electronic data associated with SH 121 and its conversion from a freeway to either a privatized or public tollway.” As to the types of ESI to be preserved, the demand is fairly well drafted and imposes a heavy burden on TxDot:

• The demand explicitly instructs TxDot to “immediately preserve potentially relevant Electronic Data including, without limitation information with the earlier of a Created or Last Modified date on or after January 1, 2004 through the date of this demand.
• The demand identifies a relatively comprehensive and quite burdensome list of potential ESI file types to be preserved, including items such as “deleted files,” “computer system activity logs,” “all file fragments and back-up files containing Electronic Data,” and “all backup tapes or other storage media.”
• The demand instructs TxDot to “preserve and retain all Electronic Data” which “relates to, mentions and/or is received or generated by TxDot… in connection with the plan and/or project to convert SH 121 into a tollway or any subsequent related work/project.” The demand specifically states that this includes all communications with: the contractor and other bidders for the project; the City of Frisco; any of numerous Texas government departments and the Legislature and the Governor.
• The demand even reminds TxDot that “you must intervene to prevent the loss of information as a result of routine operations, protocols and/or destruction policies.”

Perhaps with tongue slightly planted in cheek, the demand offers that “It is not our intent to obstruct TxDot’s everyday operation,” but then goes on to remind TxDot that “your diligent and good faith compliance of this request should also include modification or suspension of features of your information system, which in routine operation may cause the loss of relevant information….”

As with any litigation hold demand, TxDot had to make a decision: did it need to comply with the demand? If so, what is the scope of the preservation duty?

Is There an Obligation to Preserve Pre-Litigation?

The Federal Rules do not actually require the implementation of a pre-litigation hold. See Committee Note, Rule 37(f). As explained by footnote 13 in The Sedona Conference Commentary on Legal Holds (the “Commentary”):

The Advisory Committee on Civil Rules debated whether it could specify preservation obligations in the Federal Rules of Civil Procedure but ultimately decided it could not do so. Rather, the Committee opted to temper the impact of preservation obligations by protecting parties from the imposition of sanctions under the Rules for the failure to preserve certain materials in limited circumstances.

But under the circumstances here, with the City putting TxDot on notice that it was contemplating what would obviously be massive litigation and making an express demand for preservation, there really is little question that TxDot would have been taking a massive risk to ignore the City’s demand. As the Commentary puts it, The duty to preserve relevant information arises when litigation is “reasonably anticipated.” And the City’s preservation demand meets most every one of the factors listed under the Commentary’s Guideline 4, entitled, The determination of whether litigation is reasonably anticipated should be based on a good faith and reasonable evaluation of relevant facts and circumstances. These factors include:

• The nature and specificity of the complaint or threat;
• The position of the party making the claim;
• Whether the threat is direct, implied or inferred;
• The strength, scope, or value of a potential claim; and
• Press and or industry coverage of the issue either directly pertaining to the client, or of complaints brought against someone similarly situated in the industry.

Given that it would have been difficult to impossible for TxDot to treat the City’s letter as anything but putting it on notice that litigation was “reasonably anticipated,” TxDot really had no choice but to implement a litigation hold. Not so clear is what the scope of that hold should have been, and this is where TxDot, in trying to do everything right, may have shot itself in the foot.

TxDot’s Dilemma: How to Resolve an Overbroad Pre-Litigation Hold Demand

The City’s demand for all ESI “associated with SH 121” and “its conversion from a freeway,” including to and from every possible party and from many sources (such as backup tapes) that were likely inaccessible, was unquestionably burdensome for an agency as massive as TxDot. Further, according to the complaint TxDot ultimately filed, most of that information was irrelevant since (according to TxDot) the City’s claim must be brought under the Federal Administrative Procedure Act (”APA”). According to TxDot, Under NEPA and the APA plaintiffs are generally not entitled to discovery because “courts limit their review to the agency record specifically compiled for the Environmental Re-evaluation.”

Clearly this left TxDot in a quandary: if the City had initiated litigation, then TxDot could have tried to resolve these issues at the mandatory Rule 26(f) conference and, if necessary, brought a motion for protective order under Rules 26 and 34. But because this was pre-litigation TxDot did not have a clearly defined path. The path that TxDot chose is certainly hard to criticize but, as discussed below, that choice may have ultimately contributed to its failure in court.

Even though the City Refused to Clarify or Limit its Demands, TxDot Took Extraordinary Steps to Comply before It Ever Filed It’s Plea with the Court

TxDot’s initial approach was twofold: (1) TxDot made repeated pleas to the City to clarify and narrow the subject matter of the City’s demands; and (2) TxDot immediately initiated a very thorough and aggressive effort to preserve everything. Whether out of spite or an inability to clearly articulate what it really wanted, the City refused to provide any clarification and did not even respond to several of TxDot’s letters. TxDot’s exhibits to its ultimate court filing show a huge (and certainly costly) effort to preserve all ESI. For example, within three days of receipt of the City’s demand, TxDot sent an e-mail preservation notice to approximately 200 “key players”. That notice warned the key players in part that:

TxDot has an affirmative obligation to comply with the request for preservation…. This obligation requires initiation of a litigation hold to ensure relevant documents are preserved… A ‘litigation hold’ requires intervention in the routine operation…to prevent loss of information…. Please preserve any electronic edata described in the attached letter, including all backup tapes or other storage media, whether online or offline, and refrain from overwriting or deleting information that may contain the electronic data…. This includes turning off any auto delete function on Groupwise.

Shortly after, TxDot informed the key players that the preservation should include:

[I]nformation stored not only on servers, desktop computers, and laptops, but also on a personal digital assistant (PDA - i.e. iPAQ, Blackberry, Treo), flash drive or other media storage devices. Information that is stored in more than one, or even all, of these transient media must be preserved in a secure and recoverable electronic environment. If you perform official state business related to SH 121 on a home computer or any other device, that information is also subject to the litigation hold.

A short time later, TxDot published a more formal protocol for the litigation hold and sent its key players and each of its many offices detailed instructions for the preservation of e-mails and other ESI in a segregated protected network storage area as well as procedures for the restoration of documents from backup tapes. These instructions made clear that the process was being standardized and implemented in every office in the State.

Too Little Too Late: TxDot Gets Fed Up andFinally Seeks the Court’s Assistance

After about three months of this circus, TxDot apparently came to appreciate just how massive and expensive this unfettered preservation task was going to be (according to its papers, it had already spent hundreds of hours in compliance). Finally, TxDot hit on the strategy that Ralph posited last week: On August 14, 2007, TxDot filed its Plaintiff’s Original Complaint and Request for Declaratory Relief and Protective Order in the Sherman Division of the United States District Court for the Eastern District of Texas. TxDot explained the requested relief this way:

TxDot requests the court to enter a declaratory judgment ruling that the City’s letter violates the Federal Rules of Civil Procedure and is contrary to rules governing a NEPA/APA claim in federal court. TxDot requests this court’s protection from … the broad scope and undue burden of the litigation hold and requests a declaratory judgment releasing TxDot from the litigation hold as it violates the Federal Rules….

Even though the exhibits TxDot attached certainly show the burden to someone experienced in e-Discovery for all of the reasons discussed above, TxDot’s complaint did a poor job of quantifying or otherwise explaining this burden to the court. Perhaps this is part of what led the court to take a pass on this one.

The Court Dismisses TxDot’s Complaint as “Unripe”

Magistrate Don D. Bush did not treat the City’s plea kindly. Citing to Shields v. Norton, 289 F.3d 832, 835 (5^th Cir. 2002) Judge Bush noted:

A suit for declaratory relief, while allowing a party to anticipate a suit and seek a judicial resolution, must nevertheless meet this keystone limitation. In hornbook form, a declaratory action must be ripe in order to be justiciable, and is ripe only where an actual controversy exists. An actual controversy exists where a substantial controversy of sufficient immediacy and reality exists between parties having adverse legal interests. Ordinarily whether particular facts are sufficiently immediate to establish an actual controversy yields answers on a case-by-case basis. Whether a declaratory action is ripe, by its very structure, pushes against our insistence upon mature disputes. That is, it contemplates an ex ante determination of rights that exists in some tension with traditional notions of ripeness.

Judge Bush immediately hit upon the fact that it was not TxDot’s rights in the underlying environmental dispute that were at issue, but the much more amorphous concept of the scope of potential future discovery:

Notably, it is not the City’s potential claims regarding the tollway project that forms the basis of the State’s declaratory judgment action. Rather, the State seeks a declaratory judgment as to how rules of discovery and procedure might be applied by the Court to its preservation of documents in that potential suit…. As a result of the City’s litigation hold letter, the State asks this Court to determine “[w]hether it is a violation of Rules 26(f) and 34 to require an entity to broadly preserve and retain any and all electronic documents based on a required [sic] made before suit is filed.

Citing to Orix Credit Alliance, Inc., 212 F.3d at 896, 897, Judge Bush did note that a threat of litigation can indeed establish a controversy upon which declaratory judgment can be based, but he did not find that TxDot had presented such a threat since he found that the City’s letter “only states that potential exists for litigation” and not an actual threat. However, perhaps TxDot’s true failure is explained by the court’s very next paragraph:

Moreover, even if the Court were to insert itself into the pre-litigation discovery process (which it declines to do), the correspondence attached to Plaintiff’s Complaint does not evidence any concrete or developed disagreement by the parties as to the preservation of documents. A pre-enforcement action like Plaintiff’s is only ripe “if any remaining questions are purely legal … [and] further factual development” is not required for effective judicial review. The facts here are not fully developed. The dispute is abstract, “hypothetical and not suited for judicial determination.” (citations omitted)

This may be the indication of two rights making a wrong: What is Judge Bush is really saying here is that, after reviewing the exhibits, he sees: (1) a demand made by the City; (2) TxDot asking for clarification (right #1); but (2) TxDot fully complying anyway! (right #2). Since he sees a demand and the clear appearance of the ability to comply, what is the controversy? As mentioned above, TxDot does not quantify in any way the cost of its compliance efforts. All the court sees is demand and compliance, hardly the subject of a ripe dispute.

What Else Could TxDot have Done? Limited Itself to “Good Faith” Compliance.

Does this mean that Ralph’s original thesis of using a declaratory relief action to avoid an overburdensome pre-litigation demand is doomed to failure? Perhaps not. In fact, the bromide that Judge Bush offers at the very end of his decision may shed some light on the better initial course TxDot should have taken to set up the issue:

Further, while they do not specifically address pre-suit litigation hold requests, the Rules of Civil Procedure contemplate that the parties will act in good faith in the preservation and production of documents. Fed. R. Civ. P. 37. The Court encourages both parties to handle the preservation of documents in response to their respective litigation holds in such good faith. The Court declines, however, to intervene now and issue an advisory opinion as to what actions by the State would constitute good faith as to the City’s request. (emphasis added)

The court is reminding the parties that their real obligation is to act in good faith. Perhaps TxDot forgot that good faith compliance and total unquestioning compliance are not necessarily the same thing.

Part Two of the Commentary, entitled IMPLEMENTING THE LEGAL HOLD explains that:

When implementing a legal hold, it is important to recognize that the duty to preserve extends only to relevant information. While relevance is broadly defined under the Federal Rules of Civil Procedure (see Fed. R. Civ. P. 26(b)(1)), it is not without limits. As noted by one court, there is no broad requirement to preserve information that is not relevant: “Must a corporation, upon recognizing the threat of litigation, preserve every shred of paper, every e-mail or electronic document, and every backup tape? The answer is clearly, ‘no.’ Such a rule would cripple large corporations. (citing to Zubulake IV, 220 F.R.D. at 217.)

In other words, there must be some analysis of the proportionality of the potential burden with the relevance of the information sought to be preserved. Not everything necessarily need be preserved, especially if, in TxDot’s evaluation, the data was not even relevant to the anticipated dispute because either: (1) it was not part of the administrative record; or (2) it pertained to other portions of SH 121 not in dispute. While TxDot did try to engage the City in a conversation on the limits of relevance, when the City refused, TxDot simply gave in and tried to preserve everything.

Perhaps TxDot would have been better served by taking a slightly more aggressive stance. Rather than assuming there was nothing it could do without court intervention, TxDot could have responded to the City’s demand by informing them that it was so overbroad and overburdensome that TxDot would not attempt to comply but would gladly meet with the City in an effort to narrow the scope to something with which TxDot could comply. With that type of explicit notice, the City would then be placed in a quandary: either it could negotiate with TxDot (as it would have to do in litigation under Rule 26 anyway) or risk that a court would ultimately be unsympathetic with any complaint about missing documents because, after all, the City was placed on notice.

Unfortunately for TxDot,it may now have precluded itself from this lower cost option because, through its now rejected complaint, TxDot has established that in fact it can comply with the City’s request, making it less likely that a court will agree that the effort was too burdensome to impose. Thus, TxDot’s attempt to do things “right” has created the “wrong” that TxDot is stuck with a very burdensome task.

Yet another example of my favorite quote from US Congresswoman, Ambassador and playwright, Claire Booth Luce: No good deed goes unpunished.

Michael Simon

Trade Secrets Case Uses MD5 Hash and Keyword Search to Protect Defendants’ Rights - Magistrate’s Privilege Waiver Order Is Reversed

A District Court Judge in Philadelphia recently reversed a Magistrate’s order requiring a defendant in a trade secret case to produce a forensic image of two of its computers. Bro-Tech Corp. v. Thermax, Inc., 2008 WL 724627 (E.D. Pa. March 17, 2008). The computers in question were defendant’s servers located in Michigan and India. The order required production of full images to plaintiff’s counsel.

The defendant was willing to produce forensic images to plaintiff’s computer forensic expert, not its legal counsel. Defendant wanted to protect its confidential information on these servers by limiting the expert’s search to the trade secret documents, or files that might contain information about these secrets. Accordingly, defendant would only agree to allow the expert to search for files with matching MD5 hash values, matching file names, or files containing plaintiff’s keywords. Hash value searches are often used in trade secret cases. See Eg. Creative Science Systems, Inc. v. Forex Capital Markets, LLC, 2006 WL 870970, at *4 (N.D. Cal. 2006). As I explained at pages 17-20 of my article, HASH: The New Bates Stamp, 12 Journal of Technology Law & Policy 1 (June 2007), “the irreversibility quality of hashing makes it possible to perform a hash search of a computer for specific hash values without revealing the actual contents of the computer searched.”

Further, defendant was only willing to allow these searches of its servers if it could protect its attorney-client communications and work product. To do this, defendant proposed the standard procedure typically used for productions of this kind. See Playboy Enterprises v. Wells, 60 F. Supp.2d 1050 (S.D. Cal. 1999). After plaintiff’s expert performed the search of the forensic images, the files found would first be produced to defendant for a privilege review. Defendant would have a right to remove any privileged files, prepare a log of the files removed, and produce the rest to the plaintiff.

Judge Cynthia M. Rufe agreed with the defendant. She held that it was clear legal error for the magistrate to require production of the forensic images “without any limitation as to the scope of the disclosure or prior filtering for privileged or work-product materials that the images might hold.” In other words, she reversed because the order was too broad and did not protect defendant’s secrecy rights. Instead, the Magistrate erroneously assumed that the defendant had waived all of its confidentiality rights to all of the information on the servers by the mere act of having these servers examined by its forensic expert.

Case Background

Before I go into the intricacies of the waiver argument, it is helpful to review the case background. It is a trade secret action brought by Bro-Tech against one of its competitors, Thermax, and seven former employees who went to work for Thermax USA, Ltd.. The plaintiff, Bro-Tech Corporation, a/k/a “The Purolite Company,” designs and manufactures chemical solutions, namely ion exchange resins, used to remove impurities from water and air. The twenty eight page amended complaint alleges twelve causes of action:

Purolite asserts the following causes of action: (1) misappropriation of trade secrets; (2) misappropriation of trade secrets through inevitable disclosure; (3) common law unfair competition; (4) breach of contract; (5) breach of the duty of loyalty; (6) tortious interference with existing and prospective business relationships; (7) conversion; (8) violation of the Computer Fraud and Abuse Act, 18 U.S.C. § 1030; (9) commercial disparagement; (10) unjust enrichment; (11) violation of the Racketeer Influenced and Corrupt Organizations Act, 18 U.S.C. §1962(c) and (d); and (12) civil conspiracy.

Defendants responded by denying all allegations, and the competitor corporation, Thermax, counter-sued. Thermax alleged that Bro-Tech was intentionally interfering with its relationships with its customers by making false accusations that Thermax stole Bro-Tech’s trade secrets. They also claimed that Bro-Tech itself stole trade secrets, in a kind of two wrongs cancel each other out defense, known as a “clean hands” affirmative defense (it seldom works). In other words, this is a typical trade secret case with competent counsel on both sides. In fact dozens of lawyers from Philadelphia and New York have appeared of record in this case, including Baker & McKenzie for the defendants.

The amended complaint seeks, among other things, temporary and permanent injunctive relief requiring the return of any trade secrets that the individual defendants took with them or disclosed to their new employer, Thermax. Apparently to avoid a temporary injunction hearing early in the case, the defendants, in 2005, agreed to a Stipulation and Order (”the May 23 Order”) that “imposed an ongoing obligation on Defendants to return to Plaintiffs any Purolite files in their possession, and then to purge said files from their possession, custody and/or control.” Bro-Tech Corp. v. Thermax, Inc., supra at *1.

In late 2007, plaintiff deposed the defendant’s computer forensic expert, Stephen Wolfe, of the Huron Consulting Group. Wolfe testified that he had searched forensic images of defendant, Thermax’s Michigan and India servers, to see if they contained the hash values, file names, or keywords used by plaintiff’s expert, Lawrence Golden, to identify plaintiff’s trade secret files. Here is how the court described it:

Wolfe searched India and Michigan servers for (1) the unique electronic “fingerprints” (or MD5 hash values) of all Purolite documents identified as such in this litigation; (2) the file names of the identified Purolite documents; and (3) certain search terms drawn from the Golden Exhibits.

Id.at FN 8.

Wolfe admitted in his deposition that his search uncovered a number of matching files. Wolfe then filtered out files that were obviously false hits, such as standard application files that happened to contain the keywords. He then submitted the rest of the files with hits to Thermax’s legal counsel for review. Wolfe did not actually review the contents of the India and Michigan files himself, but he did review the contents of files on other Thermax computers. The court explains that:

. . . hits in the India or Michigan servers apparently were not substantively evaluated by Wolfe, but were categorized and identified according to more superficial file characteristics, filtered for “false hits” by reference to external attributes, and submitted to Thermax’s counsel for review of the actual content of the files.

Id.

The plaintiff responded to this testimony by arguing that the hits Wolfe admitted finding on Thermax’s servers in India and Michigan showed that the May 23rd Order had not been followed. The order required Thermax to return and purge any trade secrets on all of its computers. Plaintiff argued that it was therefore entitled to production of the full images of these servers and moved to compel. Magistrate Judge Carol Wells agreed after an evidentiary hearing that production was required to permit a determination of whether Defendants had violated the May 23rd Order. Judge Wells ordered the production of the full images to “designated counsel only.” Bro-Tech v. Thermax, 2008 U.S. Dist. LEXIS 8970 (Feb. 7, 2008).

Defendant appealed the Magistrate Judge’s ruling to the District Court Judge arguing clear legal error on two grounds. First, they argued:

that before any disclosure of the contents of the India and Michigan servers to counsel for Purolite occurs, Thermax has the legal right to filter the information to be disclosed in order to remove any attorney-client communications or work product material therein.

Id. at *2.

Second, defendants argued that:

they should be required to disclose to Purolite (after a review for privileged materials) only files which yield hits during a targeted search of the India and Michigan servers for evidence of Purolite files, and not, as the February 7 Order requires, to disclose the entire content of the India and Michigan servers for Plaintiffs’ counsel’s review.

Id.

Plaintiff argued that the magistrate’s order should be upheld because only inspection of the entire India and Michigan servers by Plaintiff’s counsel could ensure that no violation of the order had occurred. Plaintiff also argued that defendant had waived privilege to any confidential content on these servers “by disclosing the servers to Stephen Wolfe, who authored an expert report for Defendants, albeit one which did not, in any way, concern the content of the India or Michigan servers.” Id.

Waiver Argument

The magistrate erroneously found waiver on the basis of Rule 26(a)(2)(B), FRCP. This is the expert witness rule that requires a party to disclose all material considered by its expert in formulating an expert report to an opposing party. Plaintiff argued that this disclosure applied to all otherwise privileged materials, regardless of whether the expert actually examined the materials or relied upon them in a report. For authority, plaintiff relied upon Synthes Spine Co., L.P. v. Walden, 232 F.R.D. 460, 463-464 (E.D. Pa. 2005) (disclosure requirements of Rule 26(a)(2)(B) override all claims of attorney-client privilege), and Vitalo v. Cabot Corp., 212 F.R.D. 478, 479 (E.D. Pa. 2002) (overrides work product privilege).

Defendant countered that Wolfe had not examined these two servers as a testifying expert, but rather as a consultative expert, and so Rule 26(a)(2)(B) did not apply. Wolfe had examined and prepared reports on other computers owned by defendants, and thus was a testifying expert for these other computers. But he had not prepared a report to be used as evidence on the Michigan and India servers. Instead, he had only examined these computers to help the corporate defendant, Thermax, evaluate its case. Thus, he was only a consultative expert, and not a testifying expert, as to these two servers.

Although not discussed in this opinion, Thermax probably also argued that even if Wolfe had been a testifying witness as to these servers, and thus Rule 26(a)(2)(B) did apply, its privilege could only be waived as to specific attorney-client communications actually disclosed to Wolfe and relied upon by him to form the expert opinion stated in the report. Since Wolfe testified that he never examined the contents of any files on these servers, there was no disclosure, and, of course, no reliance.

Judge Rufe rejected the Magistrate’s over-broad construction of privilege waiver and allowed defendant to protect its privileged communications. Here is the Judge’s discussion and analysis of the law.

When privileged communications or work product materials are voluntarily disclosed to a third party, the privilege is waived. [FN18] An exception to this rule exists for disclosures to third parties which are necessary for the client to obtain adequately informed legal advice. [FN19] Under this exception, Thermax has not waived its privilege or work product protections in the India and Michigan server files disclosed to Wolfe. When searching these files, Wolfe was functioning in his capacity as “a non-testifying expert, retained by the lawyer to assist the lawyer in preparing the clients’s case.” [FN20] Thermax did not waive any protections it might have in the India and Michigan servers by disclosing them to Wolfe for consultative expert assistance in this litigation. Accordingly, this Order must provide for a privilege and work product filter.

This was obviously the correct decision, not only for the reasons stated, but also because Wolfe had only looked at information about the files (names, hash, and whether they contained key words chosen by plaintiff), and had not actually examined the contents of the files themselves. Further, only a small percentage of the files on these servers had these matching characteristics.

Holding

Here is Judge Rufe’s actual holding reversing the Magistrate’s order:

*3 In this instance, the Court must overrule as contrary to law that portion of the February 7 Order which compels Thermax to produce to Plaintiffs the entire India and Michigan servers for Plaintiffs’ review, without regard for privilege, on Rule 26(a)(2)(B) grounds. Wolfe repeatedly stated under oath that the India and Michigan servers were outside the scope of his expert report, and that he did not consider them in his testifying expert role. [FN15] Instead, his expert report exclusively concerned the contents of other devices. Because the information on the India and Michigan servers was not disclosed to or considered by Wolfe for purposes of his expert report, Rule 26(a)(2)(B) does not apply to the materials on those servers, and does not provide a legal basis for requiring their disclosure to Purolite.

Although Judge Rufe agreed with defendants that they had a right to protect their privileges, she did want a search of these servers performed to determine whether defendants had retained any of plaintiff’s trade secret information in violation of the prior stipulated order:

Notwithstanding the foregoing ruling, the Court wholly agrees with the Magistrate Judge that, in present circumstances, a significant measure of disclosure of the contents of the India and Michigan servers is necessary to ensure that Thermax has not retained Purolite information in violation of the May 23 Order. The fact that Wolfe’s electronic search of the India and Michigan servers using search terms designed to find Purolite information yielded numerous hits suggests the strong possibility (if not providing conclusive proof) that Purolite information is improperly contained in those servers. Furthermore, the parties agree that some disclosure is now necessary, although they disagree on the proper scope of the disclosure. [FN16] Thus, disclosure of the images, to some extent, shall be required.

Id. at *3.

Judge Rufe suggests that if the limited disclosure does reveal any intentional violation of the prior court order to return and purge any trade secrets, then a full search of the imaged server hard drives might be permitted. Such an inspection would include deleted files and slack space, and this might provide further evidence of intentional violation of the order or spoliation:

*4 The Court finds that there is not, at present, evidence of an intentional violation of the May 23 Order by Defendants, as would warrant full disclosure. We know too little about the contents of the files that yielded hits during Wolfe’s search of the India and Michigan servers to reach such a conclusion at this time. Wolfe’s search may have yielded false hits, or may otherwise have signaled files that were properly in Thermax’s possession; conversely, the hits may indicate a Thermax violation. Lacking clear evidence of an intentional violation, the Court will not impose the type of disclosure ordered previously in materially different circumstances involving Defendant Sachdev. Instead, a more measured, yet still significant, disclosure will be required.

Based on these findings, the court followed defendant’s suggested protocol for limited production and required the following:

*5 (1) Within three (3) days of the date of this Order, Defendants’ counsel shall produce to Plaintiffs’ computer forensic expert forensically sound copies of the images of all electronic data storage devices in Michigan and India of which Huron Consulting Group (”Huron”) made copies in May and June 2007. These forensically sound copies are to be marked “CONFIDENTIAL–DESIGNATED COUNSEL ONLY”;

(2) Review of these forensically sound copies shall be limited to:
(a) MD5 hash value searches for Purolite documents identified as such in this litigation;
(b) File name searches for the Purolite documents; and
(c) Searches for documents containing any term identified by Stephen C. Wolfe in his November 28, 2007 expert report;

(3) All documents identified in these searches by Plaintiffs’ computer forensic expert will be provided to Defendants’ counsel in electronic format, who will review these documents for privilege;

(4) Within seven (7) days of receiving these documents from Plaintiffs’ computer forensic expert, Defendants’ counsel will provide all such documents which are not privileged, and a privilege log for any withheld or redacted documents, to Plaintiffs’ counsel. Plaintiffs’ counsel shall not have access to any other documents on these images;

Conclusion

Judge Rufe has, I think, done the right thing under these circumstances. A waiver of attorney-client privilege should never be implied from a forensic expert’s mere review of a party’s computer. Otherwise, parties would be chilled from employing experts and other skillful persons to help them to evaluate a case. Would justice really be served by uneducated guesses, or blind ignorance? Do we really want to discourage clients from telling their lawyer the full story for fear that their secrets will not be safe?

It was obviously not defendant’s intent to waive its privileges in this case. The Magistrate Judge’s finding of waiver appears to have been a kind of improper punishment of defendant for its assumed violation of the prior court order. But, as Judge Rufe implies, that is taking the cart before the horse. The violation of the order has not yet been proven. The hits Wolfe testified to may all be false positives resulting from overly broad keywords by plaintiff’s expert.

In any event, even if a violation is later proven by, for instance, multiple hash value matches (which is a common way to prove trade secret theft), this would still not justify stripping defendants of their attorney client privilege. It might justify sanctions and further search of the computers. It might even result in defendant’s loss of the case on all twelve counts. But even a losing defendant has a right to communicate with their lawyer in private. It is unfair to deprive a litigant of this fundamental right as a punishment for perceived misconduct.

The United States Supreme Court has repeatedly recognized, since at least 1826, that the attorney-client privilege is a fundamental right. Public interest demands maintenance of the privilege so that a client may communicate freely and confidentially with his attorney. In Chirac v. Reinicker, 11 Wheat. (24 U.S.) 280, 294 (1826), the Supreme Court, through Justice Joseph Story, declared that “it is indispensable for the purposes of private justice” that our legal system preserve the confidentiality of facts “communicated by client to counsel” in confidence. Later, in Blackburn v. Crawfords, 3 Wall. (70 U.S.) 175, 192-193 (1865), the Supreme Court quoted with approval the following statement from an earlier English case: “If the [attorney-client] privilege did not exist at all, everyone would be thrown upon his own legal resources. Deprived of all professional assistance, a man would not venture to consult any skilful person, or would only dare to tell his counsel half his case.”

The judiciary should be wary of unwarranted intrusions upon this essential right. Judge Cynthia Rufe, like Justice Story before her, was correct to reverse the Magistrate Judge and uphold the attorney-client privilege.

“Book ‘em Danno”: Hawaiian Judge Sanctions Company for Trusting its Top Officers after One Wipes His Laptops, Allegedly to Hide Porn

A federal court in Hawaii recently imposed severe sanctions against a company for facilitating spoliation by trusting its top officers not to intentionally destroy evidence. In re Hawaiian Airlines, Inc., Debtor; Hawaiian Airlines, Inc. v. Mesa Air Group, Inc., 2007 WL 3172642 (Bkrtcy. D. Hawai’i, Oct. 30, 2007).  Defendant’s Chief Financial Officer panicked after he received a litigation hold notice and wiped files from his laptops. The plaintiff later claimed these files would have proved its case. The CFO said no, he was just trying to hide porn, but the judge didn’t believe him, and threw the book at ‘em instead.

The defendant, a regional airline company, Mesa Air Group, Inc., was sued by a bankrupt competitor for an alleged breach of a confidentiality agreement. Mesa responded by sending out a written legal hold notice. The notice instructed key players to preserve all ESI on their computers that might be relevant. Mesa timely sent out the first hold notice to its top three officers the day after the suit was filed. It trusted that they would comply with the notice and the law. It trusted that they would not act in bad faith and intentionally destroy relevant evidence.

Big mistake, according to United States Bankruptcy Judge Robert F. Faris. The defendant should not have trusted its employees, even its top officers. It should have assumed they might disobey the hold notice and the law. Mesa should have assumed its people would respond to a hold notice by destroying evidence, not preserving it. It should not only have sent out a hold notice, it should have made backup copies of the hard drives of all of its employees who might have discoverable ESI on their computers. That way, if they responded to the hold notice by deleting incriminating evidence, the company would still have a backup copy of everything to produce to the other side. (For this strategy to work the company would have to make these copies in a stealthy manner before the hold notice was sent.)

To do any less than that was, according to Judge Faris, to “facilitate” the spoliation of evidence, subjecting the company to severe sanctions; in this case, multiple adverse inferences and a fee award. The sanctions were imposed in this case even though the CFO acted alone, and there was no evidence that Mesa or its attorneys “knew of or condoned” the destruction of evidence.

According to Judge Faris, Mesa should have distrusted its Chief Financial Officer and assumed that he would destroy all relevant evidence on his three company computers (two lap tops and a virtual drive on a server) as soon as he found out about this lawsuit. On the oft chance their CFO had something incriminating to hide, and was willing to break the law to hide it, Mesa should have made copies of his various computer hard drives, and not simply relied upon a written notice. In Judge Faris’ words: 

13. Mesa could have taken reasonable steps that would have prevented, or mitigated the consequences of, Mr. Murnane’s destruction of evidence. For example, Mesa could have made a backup of Mr. Murnane’s H drive and the hard drives of Laptop 1 and Laptop 2 promptly after HA filed suit. Doing so would not have been costly, burdensome, or unduly disruptive of Mesa’s business. Instead, Mesa simply told Mr. Murnane to preserve all evidence and trusted him to comply. Even though Mr. Murnane was a valued, trusted, high level employee of the company, Mesa could and should have taken reasonable steps to prevent all of its employees from doing wrongful and foolish things, like destroying evidence, under the pressure of litigation. Because Mesa failed to take such steps, Mesa facilitated Mr. Murnane’s misconduct.

It is true that the imaging of two laptops and a virtual drive would not, in and of itself, have been terribly burdensome or expensive. But does that justify the mandatory stealth imaging of all impacted employees? Mesa is an airline of over 5,000 employees, generating revenues of over $1.4 billion per year. The opinion states that only three employees were sent the original preservation hold notice, the three top officers of Mesa: the CFO, CEO and the President. If in fact only three custodians were involved throughout, which to me seems unusual, then the court’s low cost and burden argument has some merit, even if it is still questionable on policy and practical grounds. But if other hold notices were later sent out to dozens of additional ESI custodians, which to me seems more likely, then the court’s economic analysis is flawed.

The policy of mistrust is also, in my view, not well considered. Although hindsight is 20/20, how was Mesa to have known when suit was filed that its Chief Financial Officer might destroy evidence? There is nothing in the opinion to suggest he was anything other than a trusted and reliable senior management employee. If Mesa could not trust its Chief Financial Officer, then it could not trust anybody. By this logic, in every lawsuit Mesa would have to image the computers of all key witnesses and ESI custodians who might have discoverable ESI in them. Any of them might do “wrongful and foolish things” under the pressure of litigation. Where would this lead? I am reminded of the quote by Ralph Waldo Emerson: “Our distrust is very expensive.” The copying of dozens, if not hundreds, of computers can become very expensive. Is it really reasonable to expect large organizations to always act out of mistrust and fear that it might have a renegade employee, one who is willing to break the law and destroy evidence? Is it really fair to hold that an employer facilitated its employees’ bad faith destruction of evidence, simply because it did not copy all potentially impacted computers as soon as a suit was filed?

Also, as a practical matter, how was Mesa supposed to have copied its top officers’ computers before they had notice of the law suit, and thus an opportunity to delete files from these computers? They are the first ones to learn of a suit like this, and are necessarily involved in discussions with the lawyers on what to do, who should be provided with a hold notice, and the like. Is it realistic to require legal counsel to copy everything on the computers of the top officers of a company any time a suit is filed where they might be involved as a witness? Should in-house counsel be required to do so surreptitiously, even before the officers are told about the lawsuit? I doubt they would last very long if they did! Such extreme measures should, in my opinion, only be employed in very rare circumstances where there is strong evidence that the action is required, that otherwise there is a substantial likelihood evidence will be destroyed. Even then, it should be used with caution. The extreme process of imaging all computers should never be used in a case such as this, where there is no advance warning of possible spoliation, much less a strong showing of likely destruction of evidence.

Beyond the questionable holding, the facts underlying this case are interesting on a number of levels, including the technical “geek” perspective. Mesa’s CFO was said to be “an experienced and knowledgeable computer user.” He installed a program called System Mechanic Professional 6 on both of his company owned laptops, and used it to super-delete files from them. He attempted to disguise the timing of these deletions by changing the dates on the computers before he ran the software. He did not know that forensic analysis can easily detect such system clock changes. The CFO also stored files on a company server, called his “H drive,” but he did not use special software to super-delete any files there. Some files on his “H drive” were deleted, and apparently could not be later restored due to normal usage, but were recovered from backup tapes of the server. These and other technical details are explained by the Court:

System Mechanic has a subprogram called DriveScrubber2 that permanently deletes files from a computer.

7. When an active file (a file that a user can view) is deleted on a computer using the Windows operating system, the data comprising the file is not erased; instead, the file is removed from the “index” of all active files on the disk, and the disk space that contains the deleted file is gradually written over as new files are written to the disk. A person with the appropriate skill and software can analyze the bits of data left on the “unallocated space” of the drive (the portions of the disk that, according to the “index,” do not contain active files) and reassemble some or all of the deleted files. “Disk wiping” programs like DriveScrubber2 render deleted files unrecoverable by writing meaningless data (usually repeated strings of hexadecimal characters) to the unallocated space of the disk, permanently eliminating the residue of previously deleted files.

Later forensic exams of the CFO’s laptop computers could not recover the  super-deleted files, but they did prove that Drive Scrubber was used after the hold notice was received. The exams also revealed a clumsy attempt to conceal the disk wipes by changing the system clock to a time before the hold notice. As to the files ordinarily deleted from the network server “H drive,” Mesa was able to recover “many, if not all” of them by restoring the backup tapes of that drive. Some of the restored files were relevant, but none were “smoking guns.” 

The CFO did come up with a creative defense to his actions. He claimed he was trying to hide the fact that he had been viewing “adult materials.” Judge Faris did not believe this testimony. For one thing, Mesa’s CEO, who the judge said was a good friend of the CFO (and thus obviously trusted him), testified that his friend had ”told him he had wiped the hard drives in order to conceal adult content on his computers.” Still, according to Judge Faris:

It was absolutely clear from Mr. Ornstein’s words and demeanor on the witness stand that he did not believe Mr. Murnane’s “adult content” explanation.

I do not know what the CEO’s demeanor was like when he testified about his friend’s excuse - no doubt he was embarrassed - but obviously the CEO believed him or he would not have offered the story. Also, the opinion admits that evidence was offered that the CFO’s laptop was previously found to have adult content in two prior incidents in 2003 and 2004. But Judge Faris was more impressed by the facts that no adult materials were found on the laptops now, nor on the “H drive”, and the IT tech who “regularly worked” on the laptops testified that he had never seen adult content on the computers. 

Also, Judge Faris commented that the CFO never took the stand to testify, and he found that:

It is not credible to suggest that a high-level officer and busy person such as Mr. Murnane would have done a mundane task like this himself rather than leaving it to Mesa’s IT department.

That does not have the ring of truth to me. If a person, no matter what his rank, has the time and indiscretion to view porn on his company laptop, then it does not surprise me that he also has time to wipe it clean, and would not want talkative IT employees to do it for him. No one, especially a “high-level officer,” would want to entrust such a sensitive, and potentially embarrassing task to the company’s IT department. The super-deletion process is not really too hard or time consuming, as the judge here seems to recognize by calling a “mundane task.” So it is not at all surprising that someone would want to do it themself, much less “not credible.”

The timing of the CFO’s deletions of alleged “adult materials” was too suspicious. There were also several indications that the CFO had taken and used confidential information of the bankrupt Hawaiian Airlines, and thus Mesa had breached the confidentiality agreement as alleged. These other facts seemed to color the court’s analysis of the spoliation motion and conclusions that:

b. The confidentiality agreement provides that “Mesa shall be responsible for any breach of this agreement by Mesa’s employees, officers and Representatives….” Mesa should also be responsible for the intentional destruction by one of its highest ranking officers of evidence that could have shown whether Mesa complied with that agreement.

c. Mesa could have prevented Mr. Murnane from destroying evidence, or at least limited his ability to destroy evidence, by taking reasonable, inexpensive, and non-burdensome steps. Mesa failed to do so and is responsible for the consequences of that failure.

For these and other reasons, Judge Faris granted plaintiff’s motion for sanctions, imposed multiple adverse inferences and taxed fees, but stopped short of entry of a default judgment as plaintiff had requested.

Sherlock Holmes in the Twenty-First Century: Definitions and Limits of Computer Forensics, Forensic Copies and Forensic Examinations

If Sherlock Holmes were alive today, he would surely be a master of  computer forensics. Just as he sometimes used his chemistry set in the 19th Century to analyze clues, today he would use forensic software to examine digital devices. Holmes would know how to make forensic copies of computers, i-phones, thumb-drives and other ESI storage devices, and also know when not to waste his time doing so. No doubt Dr. Watson would be amazed at the evidence Holmes would sometimes uncover. The forensic examination of computers is an important tool in twenty-first century detective work, but it is no panacea. Sherlock Holmes of all people would know that it is not a substitute for clear thinking and rational deductions, and is not appropriate in every case. 

Lots of trial lawyers do not really understand computer forensics, and are prone to think that a full scale forensic examination of all computers is needed in every case. They want their tech-guys to make ”forensic copies,” work their mumbo-jumbo on each, and like Sherlock Holmes, come up with an amazing and unexpected clue that solves the case. Sometimes this fantasy comes true, but only rarely. The attempt to search every bit and byte of every computer, including the deleted files and slack space, is expensive.  Most experts agree that this kind of “deep dive” forensic examination work should be done sparingly, and is not needed in most e-discovery cases. Even when a special case suggests it may be needed, such forensic exams rarely produce the killer email that wins the day. The lawyer who uses this kind of full scale forensics approach in every case is setting himself up for major disappointments and wasting his client’s money.

What is ”computer forensics,” and the related terms, ”forensic copy” and “forensic exam”? Let’s begin by defining “forensic copy,” which is fairly simple.  A forensic copy is an exact bit-by-bit copy of the entire physical storage media, including all active and residual data and unallocated space on the media. This is also sometimes called an “image copy” or “mirror image.” See The Sedona Conference Glossary: e-Discovery & Digital Information Management, The Sedona Conference Working Group Series, May 2005.

A forensic copy allows for a “forensic exam” of the copy. You do not examine the original because the act of examination would, in itself, change the original. (This is called the Heisenberg Principle of computer forensics.) In a forensic exam, all of the information on a disk is carefully probed and searched, even the otherwise hidden information: the deleted files, residual data, unallocated space, corrupted files, encrypted files. In a forensic exam, everything that is scientifically possible to restore and search is searched, including ESI classified as not-reasonably-accessible under Rule 26(b)(2)(B).

The definition of the more general term “computer forensics” is more challenging. It is not a specific procedure like forensic copy or exam, it is an entire field of study or scientific discipline. The National Institute of Standards and Technology special publication (SP) 800-86 Guide to Integrating Forensic Techniques into Incident Responses provides an authoritative definition of computer forensics:

. . . the application of science to the identification, collection, examination, and analysis of data while preserving the integrity of the information and maintaining a strict chain of custody for the data. Data refers to distinct pieces of digital information that have been formatted in a specific way.  . . .

The NIST explains how the process of computer forensics has four basic phases:

Collection: identifying, labeling, recording, and acquiring data from the possible sources of relevant data, while following procedures that preserve the integrity of the data.

Examination: forensically processing collected data using a combination of automated and manual methods, and assessing and extracting of particular interest, while preserving the integrity of the data.

Analysis: analyzing the results of the examination, using legally justifiable methods and techniques, to derive useful information that addresses the questions that were the impetus for performing the collection and examination.

Reporting: reporting the results of the analysis, which may include describing the actions used, explaining how tools and procedures were selected, determining what other actions need to be performed (e.g., forensic examination of additional data sources, securing identified vulnerabilities, improving existing security controls), and providing recommendations for improvement to policies, procedures, tools, and other aspects of the forensic process.

A well known IT site, SearchSecurity.com, provides another good definition of computer forensics:

Computer forensics, also called cyberforensics, is the application of computer investigation and analysis techniques to gather evidence suitable for presentation in a court of law. The goal of computer forensics is to perform a structured investigation while maintaining a documented chain of evidence to find out exactly what happened on a computer and who was responsible for it.

Forensic investigators typically follow a standard set of procedures. After physically isolating the computer in question to make sure it cannot be accidentally contaminated, investigators make a digital copy of the hard drive. Once the original hard drive has been copied, it is locked in a safe or other secure storage facility to maintain its pristine condition. All investigation is done on the digital copy.

Investigators use a variety of techniques and proprietary forensic applications to examine the hard drive copy, searching hidden folders and unallocated disk space for copies of deleted, encrypted, or damaged files. Any evidence found on the digital copy is carefully documented in a “finding report” and verified with the original in preparation for legal proceedings that involve discovery, depositions, or actual litigation.

The Sedona Conference Glossary also defines computer forensics:

Computer Forensics (in the context of this document, “forensic analysis”) is the use of specialized techniques for recovery, authentication and analysis of electronic data when an investigation or litigation involves issues relating to reconstruction of computer usage, examination of residual data, authentication of data by technical analysis or explanation of technical features of data and computer usage. Computer forensics requires specialized expertise that goes beyond normal data collection and preservation techniques available to end-users or system support personnel, and generally requires strict adherence to chain-of-custody protocols.

A recent commentary by forensic expert, Ken Zatyko, in Forensic Magazine focused on the difficulty of defining what he called “digital forensics,” which for purposes of this article, I consider equivalent to “computer forensics.” Ken Zatyko is a recently retired Air Force Lt. Colonel who was the director of the Department of Defense Computer Forensics Laboratory for many years, and is now an Adjunct Professor with John Hopkins University. Ken reviews several other definitions as I have done, and then settles on his own definition that he urges others to adopt:

The application of computer science and investigative procedures for a legal purpose involving the analysis of digital evidence after proper search authority, chain of custody, validation with mathematics, use of validated tools, repeatability, reporting, and possible expert presentation.

This is the best definition I have seen, and my personal favorite, perhaps because it includes “validation with mathematics,” a reference to my favorite subject in computer forensics, hash analysis (See the Blog Page above, HASH, and my law review article on this subject: HASH: The New Bates Stamp).  Zatyko then goes on to delineate an eight-step forensics process:

1. Search authority
2. Chain of custody
3. Imaging/hashing function
4. Validated tools
5. Analysis
6. Repeatability (Quality Assurance)
7. Reporting
8. Possible expert presentation

The various definitions make clear that ”computer forensics” is a disciplined, scientific approach to electronic discovery and evidence validation.  Computer forensics in this general sense should be followed whenever electronic evidence is involved in a legal proceeding, which in today’s world means almost every case. In that sense, the trial lawyer may need a person familiar with computer forensics on every case to supervise e-discovery activities. Trial attorneys must be able to verify that proper procedures, authenticity and chain of custody were followed in order for the ESI discovered to be admissable as evidence at trial. This is, however, a far cry from a full scale Sherlock Holmes forensic examination of all computers.  It is important for attorneys to understand the difference between forensics as a general discipline to lay a proper predicate for evidence, and forensic copying and forensic examinations as particular applications of this discipline, applications that are not necessary in every case.

One person who has a good grasp of this difference is John Patzakis. He is the General Counsel of Guidance Software, makers of EnCase, the forensics software tool used by over 80% of computer forensics experts. Although it might be tempting for him to push the over-use of forensics, he does not do so.  He and his company are a class act, which is one reason I am pleased that John agreed to do a West-Thompson Webinar with me later this year: ”Computer Forensics and E-Discovery.” We will be joined by another e-discovery attorney, a modern-day Sherlock Holmes of computers, Bill Speros, who also understands this distinction very well, and by a well-known accountant forensics expert, Frank Wu of Protivity. 

John Patzakis was interviewed in 2007 by Forensic Focus, a website for “computer forensics news, information and community.” John’s interview provides some good advice on the prudent and restrained use of computer forensics in e-discovery. 

In general, eDiscovery tends to involve a “computer forensics-like” approach, if you will, where aspects of traditional forensics such as chain of custody, metadata recovery and preservation, documentation and reporting and an overall defendable process are central requirements. Aspects of traditional forensics that are generally not as important include full disk imaging, deleted file and file fragment recovery, and deep dive analysis involving various artifacts.

This reference to ”traditional forensics” is what most people think of when they hear “computer forensics,” the expensive CSI-type criminal investigations where computer disks are imaged and forensic exams are performed to restore and search deleted files, fragments, Internet cache, slack space, memory, and the like.  A diagram providing a simple overview of the forensic examination process using EnCase software is shown below.

John Patzakis has written a very comprehensive treatise on electronic discovery law related to his company’s software tools and forensic related issues called the EnCase Legal Journal  (April 2007). At 143 pages and 446 legal citations, this is not your typical vendor white paper, and is well worth reading and using as a reference. Section 9.5 of the Journal is entitled ”Cost-Effective Searching of Data.” It pertains to my original point that many trial lawyers tend to over use computer forensics and seek full-disk imaging and other “deep-dive” analysis in every case.

Collection and preservation of ESI must incorporate a defensible process that accomplishes the objective of preserving relevant data, including metadata, and establishing a proper chain of custody. With the right technology, these results can be achieved without full-disk imaging. However, full-disk imaging and deleted file recovery are emphasized by many eDiscovery vendors and consultants as a routine eDiscovery practice. While such deep-dive analysis is required in some circumstances, full-disk imaging is unwarranted as a standard eDiscovery practice due to considerable costs and burden. Large-scale, full-disk imaging is burdensome because the process is very disruptive, requires much more time to complete, and, as eDiscovery processing and hosting fees are usually calculated on a per-gigabyte basis, costs are increased exponentially. . . .

Generally, courts will only require that full forensic copies of hard drives be made if there is a showing of good cause supported by specific, concrete evidence of the alteration or destruction of electronic information or for other reasons. Balboa Threadworks, Inc. v. Stucky, 2006 WL 763668, at *3 (D. Kan. 2006); However, “[c]ourts have been cautious in requiring the mirror imaging of computers where the request is extremely broad in nature and the connection between the computers and the claims in a lawsuit are unduly vague or unsubstantiated in nature.” Ameriwood Industries, Inc. v. Liberman, 2006 WL 3825291, (E.D. Mo. Dec. 27, 2006).

I wrote about the Ameriwood case in my blog, Employer Allowed to Mirror Employees-Home Computers and Obtain Inaccessible ESI. Ameriwood was one of the first decisions in the country to employ the new inaccessibility analysis under Rule 26(b)(2)(B). Although the court in Ameriwood was cautious, it decided to allow the employer to make a forensic copy of the employee’s computer, and search for otherwise inaccessible ESI, the deleted files and slack space.  The court only allowed this kind of forensic imaging because the employer had made a special showing of good cause under Rule 26(b)(2)(B). The general rule is to be cautious and not allow such forensic exams absent a showing of good cause. Good cause can come in a variety of forms, but usually arises from suspicious circumstances that suggest spoliation, such as a story of a midnight hacker erasing all of your files, or the loss of a laptop with all of your records just before a deposition duces tecum.

In another case, Hedenburg v. Aramark American Food Services, 2007 U.S. Dist. LEXIS 3443 (W.D. Wash. Jan. 17, 2007), the court applied the general rule and denied the application for a forensic exam. The employer requesting the forensic imaging did not provide good cause as required under Rule 26(b)(2)(B). I wrote about Hedenburg in my prior blog Forensic Fishing Expedition Rejected. This is an employment discrimination case where the employer wanted a forensic copy made of the employee’s personal computers. The employer proposed that the copy then be examined by a computer forensic expert serving as a special master. The employer’s attorneys had an expansive view of computer forensics not warranted by the facts or the law. 

In a move reminiscent of Inspector Lestrade, employer’s counsel provided no good reasons for the exam, and instead argued that such exams were common in these types of cases, and might lead to important clues. The Judge rejected the proposed forensics as a mere “fishing expedition.” Blind hope may be a fisherman’s credo, but it will not work in court, and is no substitute for the kind of cold logic and reasoned analysis made famous by Sherlock Holmes.

For more information on forensics check out the audio CLE I did for West Legalworks entitled: E-Discovery and Computer Forensic Investigations 101: When Does Your Case Warrant the Full “CSI” Treatment? With me on the panel for this 1.5 hour webcast were J. William Speros, Consultant and Principal, Speros & Associates LLC; Michael Michalowicz - Associate Director, Protiviti; and, John Patzakis, - Vice Chairman and Chief Legal Officer, Guidance Software.

Should You Save Internet Cache?

Does the duty to preserve potential evidence require you to save your Internet cache? A district court in Pennsylvania recently addressed this issue, and, indirectly at least, said NO. Healthcare Advocates, Inc. v. Harding, Earley, Follmer & Frailey, 2007 WL 2085358 (E.D. Pa. June 20, 2007). The court held that the defendant’s automatic and unwitting deletion of cache files did not constitute spoliation, and did not warrant any kind of sanctions, even though potential evidence had been destroyed.  The court did not squarely hold there was no duty to preserve Internet cache per se; instead, it held that, in this case, the destruction of evidence contained in the temporary cache files was accidental, and was not prejudicial, so no sanctions were appropriate.

In my opinion, the court here got it right, and in most cases there is no need to take the time and effort required to preserve cache files. Still, there may be rare exceptions to this general rule where you should save cache; for instance, if an employee is fired for viewing porn-sites at work and then he or she immediately files suit.  In this circumstance, Internet cache files would provide critical evidence, and the custodian of the computer should save the cache.  Assuming the employer knows about Internet cache, then they would want to preserve the cache, because it proves what websites have been visited on a computer.  Of course, if the employer does not know about cache files, and how they can be automatically deleted, much like the law firm defendant apparently claimed in the Healthcare Advocates case, then they would not know to save their cache.  Can you breach a duty to preserve evidence that you did not even know existed, much less was in danger of destruction?  That is a difficult question that was not really answered by this case.  The answer depends on whether you think the party, or their lawyer, should have known about cache files, such that it was negligence, perhaps even gross negligence, not to preserve these mystery files.

Most of the readers here will know what Internet cache files are. But some may not be sure, so let’s start with a wee bit of technical background.  Internet cache is not a misspelled ”Pay-Pal” money thing, it’s a temporary storage area where frequently accessed data are placed for subsequent rapid access.  It can be RAM memory type cache, or it can be cache that is written to a drive.  The Internet cache files we are talking about here are of the latter variety.  Microsoft calls this kind of cache ”Temporary Internet Files.”

As you know, when you “go to” an Internet website, the website is actually coming to you.  Copies of the website files are transferred from the web server computer to your computer. These files are actually downloaded and saved to your computer hard drive.  (There are of course some exceptions to this for some website content, such as streaming video and the like, but as a general rule this is how it works.) These html and related web files are then viewed with your browser software, such as Microsoft’s Internet Explorer, from your own computer. The place where the downloaded web files are stored on your computer is called the “Internet cache.” Many people do not know that when they leave a website, and go someplace else on the web, the website itself remains lurking and hidden on their computer. 

The Internet cache web files are stored on your computer because you may return to that same website again.  If you do, and you point your browser to the same address, the browser will know that it already has the web in its cache, and so will not actually go out and fetch it again from the remote webserver. That could take some time. The browser will instead just display the web files you have already visited and stored in your cache.  It is much faster that way, which is the whole point of a cache. Of course, in the meantime, the web may have changed, and you may not know it because you are looking at the web you previously downloaded.  No worries, however, because you can defeat that, and force the browser to go out and get the file again from the webserver, by simply using the refresh file function.

You may wonder where these Internet cache files are stored; for if you do not know that, it is hard to preserve them. Assuming you are using Explorer, go to the ”Tools” menu at the top, and select “Internet Options.”  The first “General” page will have a “Browsing History” section. In Internet Explorer 7, you have the option to “Delete temporary files, history, cookies, saved pass words and web form information.”  If you hit the “Delete” button you will erase all of the Internet cache files, the history of the webs you have visited. To the right of the “Delete” button is the “Settings” button.  Go to the “Settings” page (the Explorer 6 version of this page is shown in the graphic below)  and you will find choices for storing Internet files, meaning how often the browser will check to see if the web page has changed from the version stored on your disk.  The Settings page also shows where these files are currently located. That is the location of your Internet cache, a/k/a your Temporary Internet Files folder.  It will also allow you to specify how much space you want to allocate for the cache.  When that limit is filled it will start writing over the old cache files.  You can choose from between 8 to 1024 megabytes.  Many people max out to the 1024 limit because space is plentiful and cheap, and the more cache the faster your web browsing.

The default location for the Internet cache is usually something like what is shown above: “C:\Documents and Settings\v-megans (the user’s name)\Local Settings\Temporary Internet Files\”.  But you can change that to be any location you would like. On this same menu, you can also choose to view these files.  At the bottom of this page, Explorer allows you to specify how many days to save “the list of websites you have visited.” This is not a full cache of the websites visited, just a list of the addresses. 

The above summary is not intended to be complete; there can be many variances depending on browser software and other configurations.  In addition, other files are cached when the Internet is used, most especially for email, where copies of all emails sent and received may be stored on disk in other locations depending on the software utilized.

Back to the Healthcare Advocates case: it involves some very interesting facts, albeit a farfetched complaint.  I do not have all of the facts underlying this law suit (some are filed under seal), but from the court’s opinion, it appears that this is little more than a sour grapes type of spite suit.  To understand this case, you have to understand the one that preceded it.  There the same plaintiff, Healthcare Advocates Inc. (hereinafter “HAI”), sued a competitor for trademark infringement and trade secret misappropriation.  HAI lost this case, in large part because of excellent lawyering by the defendant’s attorneys, the law firm of Harding, Earley, Follmer & Frailey (hereinafter “Harding”).  Harding used the very handy service found at archive.org called the “Wayback Machine.” If you are not already familiar with this web and service, I suggest you check it out.  The archive’s Wayback Machine allows you to view prior versions of websites that have been saved by archive.org.  It is a very good way to determine what trademarks and other materials were actually in use by a company in the past.  Harding used it to find out what HAI’s website had looked like in the past.  The old versions of HAI’s web proved to be very powerful evidence against HAI, and they lost their case by summary judgment. 

So what did HAI do next?  They filed yet another law suit, this time against their competitor’s attorneys, the Harding law firm. Unbelievable, but true! What horrible things is the law firm alleged to have done? HAI accuses them of violating their copyrights by “hacking” the Wayback Machine so as to download old versions of their website! Never mind that there was no evidence of hacking, and Harding only used the archive.org repository the way any good law firm would: to find the truth and defend their client.

To add insult to injury, HAI tried to dress up this second case with charges of spoliation based solely on Harding’s failure to preserve the Internet cache files of the old HAI’s web pages.  HAI even had the audacity to argue that the law firm should have stopped using their computers altogether so that the temporary cache files would not be lost.  In their losing cross-motion for summary judgment, HAI’s lawyers got carried away, again, and argued that Harding’s failure to stop using their computers to preserve the Internet cache was such a bad act of spoliation that it ”shocks the conscience.” Here is the understated way the court responded to these arguments:

The Harding firm had no reason to anticipate that using a public website to view images of another public website would subject them to a civil lawsuit containing allegations of hacking.

Thus, the failure to immediately remove computers that the firm used every day, when they had no reason to believe that their actions would subject them to a lawsuit for “hacking,” is not an action that shocks the conscience.

I am pleased to report that this second suit has been dismissed too, again by summary judgment. Obviously it was a “fair use” by the law firm of the discontinued copyrighted web pages, and not even close to a copyright infringement.  Moreover, the spoliation charges were just as bad.  It is not clear from the opinion, but I think that Harding knew full well about Internet cache files.  It is after all a group of Phildelphia lawyers specializing in intellectual property law. Harding simply choose not to preserve these files because there was no reason to save them.  They had already printed out all of the webs, and used them in evidence. Why should they also preserve the cache?  The whole suit by HAI is just plain bizarre. The court agreed, and found no prejudice at all to the plaintiff from the deletion of the cache files.

There are two more things that surprise me about this case. First, Internet Archive, the non-profit group behind www.archive.org,  was joined as a defendant to the case.  That is not surprising given HAI’s obvious litigiousness, but it is hard to understand why the archive group settled with HAI, instead of moving for summary judgment. Apparently, they were concerned because their exclusion policy, which supposedly allows any website to opt-out of the archive and its Wayback Machine by the use of a robots.txt file, did not work in this case.  HAI had, for obvious reasons, tried to have its web excluded from the archive before it filed the first law suit against its competitor, but the exclusion failed and Harding was able to get at the truth. The terms of the settlement are confidential, so we can only speculate why Internet Archive preferred to settle.  The second surprising thing is that the Harding firm provided HAI with a forensic image of their computer’s hard drives. It impressed the court that Harding had nothing to hide, but one wonders why they bothered.

Although not reported in the decision, my review of the docket sheet for this case shows that there is now a motion pending by Harding for an award of $161,461.50 in attorney fees. They argue that this is a frivolous case, and an award of fees in this amount, plus costs of $9,348, is justified. 

IT Tech’s Fast-Talk Had Zero Persuasive Value with Judge

A district court judge in Connecticut recently rejected defendant’s explanation as to why the hard drives of key employees contained only zeros, and imposed sanctions for spoliation.  Jane Doe v. Norwalk Community College, 2007 WL 2066496, 2007 LEXIS 51084 (D. Conn. July 16, 2007).  

First some background of this “Jane Doe” case. The main defendant here is a state community college.  The plaintiff is a student alleging her college was negligent in its retention and supervision of a professor who sexually assaulted her.  The now “former professor” is also a defendant, but with no legal representation.  The student was permitted to file suit as ”Jane Doe” to protect her privacy. 

After two years of litigation, Jane Doe persuaded the court that the college was withholding electronic evidence.  The school was ordered to produce the computers of key witnesses for inspection by Doe’s computer forensic expert, Dorran Delay of DataTrack Resources.  The expert inspected the college computers over a two day period. Here is where the case gets interesting. Delay’s inspection showed that several of the computers had no data: they were literally all zeros.

Jane Doe’s next move was to file a motion for sanctions based on spoliation of evidence.  She alleged that “the hard drives of key witnesses in this case were scrubbed” or “completely ‘wiped’ of data.” This led to a flurry of affidavits by Doe’s expert, Delay, and the counter-expert used by the college, its own in-house Information Technology Technician, Wyatt Bissell. Of course, the experts did not agree. Bissell came up with a laundry list of excuses for why two computers were “full of nothing.” He tried saying it was the wrong computer, then that it was not wiped at all, just imaged. Then, as a last resort, he settled on the best excuse of many an IT Tech, that the “all zeros” problem was simply the result of “computer failure.”

The judge responded by scheduling two evidentiary hearings.  At these hearings, Delay, Bissell, and other witnesses testified and were cross-examined about the many suspicious circumstances surrounding the missing ESI.  Further, at one of the hearings, the college offered the expert testimony of another of its employees, Mr. Olsen, the Information Technology Systems Manager.  It did not help much.  Among other things, both Bissell and Olsen testified that they did not think the state’s two-year document retention policy applied to them or “normal computer usage,” directly contradicting the hearing testimony of their boss, the Dean.  The testimony of the defense experts was rejected by the court as not credible, and overall, they only served to make a bad situation worse.

District Court Judge Janet Hall not only rejected the defense expert testimony, she rejected the legal arguments of defense counsel as well.  One of the more clever arguments they made, to no avail, was that they could not put an effective hold in place without revealing the true name of Jane Doe. Judge Hall said they should have contacted plaintiff’s counsel and tried to work that out.  Defense counsel’s arguments as to when the duty to preserve commenced were also given zero value.  It seems as if the attorneys’ credibility was completely nullified by the specious testimony of their experts. 

In the end, Judge Hall granted Jane Doe’s motion, and awarded an adverse jury instruction based on the grossly negligent failure of the college to preserve ESI.  She also awarded Doe her expert witness’s costs, which, I suspect, will be quite large. 

In a case like this an adverse inference instruction is almost always fatal to the defense.  For all practical purposes, even though the case has not yet been tried, it has already been lost because of e-discovery. The only real question still remaining has to do, once again, with zeros.  How many will be added to the judgment or settlement?

Although this is all well and good, to me the most interesting aspects of this case are its computer forensic, geek-type technicalities.  First of all, the forensic expert, Delay, and the college IT technician, Bissell, could not agree on whether the computers had been ”wiped.”  Delay opined that the “all zeros” condition of the hard drives showed that they had been intentionally wiped or scrubbed of all data.  Footnote 3 of the opinion explains that:

According to Delay, wiping is a “process that overwrites existing data on the hard drive, making this information unrecoverable.”

 Bissell’s counter explanation is set forth in footnote 6: 

At the Hearing, Wyatt Bissell indicated that he disagreed with the term “scrubbed,” which overwrites a hard drive, completely eliminating all data from it. Instead, Bissell testified the correct word to use is “imaged”–that is, NCC’s [the college] technology modifies the structure of the hard drive, without scrubbing it.

Bissell also testified:

. . . that Delay’s results, i.e., that it appeared that this particular hard drive had been “scrubbed” were because Schmidt’s hard drive was in the process of failing, which can produce inconsistent or corrupt results.

The court did not believe Bissell and found that the computers had been “scrubbed’ or “wiped.”  Judge Hall explained what she meant by these terms in footnote 11:

By “scrubbed” or “wiped” the court means more than overwriting or “reimaging;” it means eliminating all data from the hard drive, such that none of the old data can be read or still remains on it.

It is hard to see how you can reach any other conclusion when presented with a computer hard drive filled with all zeros. That is what most (but not all) data scrubbing programs are designed to do. (For an example of one such program, GhostSurfer, see my blog of June 7, 2007, GhostSurfer Wipe Out Leads to Jail Order Sanction in Bankruptcy Court.)  Most data erasure software physically writes zeros (or ones, or random combinations) to all sectors of a hard drive and thereby completely writes over and erases everything, even residual data existing outside of any organized file structures.  This process is also known as “shredding,” and among Mac users is called “zeroing all data.”  Supposedly there is expensive equipment available that allows for the recovery of segments of a hard drive even after it has been zeroed out.  For that reason, many data shredding programs provide for multiple wipes with various types of random patterns of data filling.  This will defeat even the spy agencies who own such equipment, and so meets the Department of Defense specifications for destruction of sensitive data.  (The really top secret stuff is physically destroyed, cut up into tiny bits (no pun intended), and then dumped into multiple land fills.)

To better understand how this kind of disk wiping works, you need to recall that all computers operate and store information in bits of either one or zero, electrically on or off.  This is the binary code.  Recall also that eight of the on-or-off bits together comprise a byte.  A typical hard drive today has hundreds of billions of bytes.  Thus if a hard drive, or any other ESI device, contains all zeros, or all ones for that matter, it contains absolutely no information at all.   Information can only be stored when both ones and zeros are used in the almost innumerable possible permutations.  This all-zero condition does, however, tell you that the disk has been intentionally wiped.  Contrary to Bissell’s testimony, a computer which has been imaged, or is subject to failures of some kind, would not contain all zeros.  Some information, some combinations of ones and zeros among the billions of bits on a hard drive would remain.  Judge Hall explains how this applies to the case as follows:

Delay found that it contained all 0’s, indicating that every sector had been overwritten. Delay testified that, if the drive had data on it but was failing, as Bissell testified, then data would be seen on it with Delay’s forensic software, which instead recognized that the hard drive was unpartitioned and contained no data. Moreover, Seaborn’s new computer had traces of other users’ information on it, thus showing an inconsistent result in NCC’s process of re-imaging hard drives. Even if it was consistent with NCC’s policy, the fact that Seaborn’s new computer showed other users’ information indicates that “imaging” does not eliminate everything from a hard drive, but leaves some data from old users on it, prompting the question why Seaborn’s old computer–or Schmidt’s computer–did not have any evidence of other users on it. The answers provided by the defendants–a failing drive or “re-imaging”–are rejected by the court as not credible. 

The irregularities in PST files that Delay uncovered are another factor worth mentioning that led Judge Hall to suspect that relevant evidence had been intentionally destroyed by several of the college employees.

Additionally, Delay found the Microsoft Outlook PST files, which house electronic mailboxes, of four individuals had inconsistencies “that indicate [ ] that data has been altered, destroyed or filtered.” Id. at ¶ 6. For example, Professor Skeeter’s PST file contained no Deleted Items and only one Sent Item and the Inbox and Sent Items contained data starting August 2004, “even though other activity is present starting in 2002.” Id. at ¶ 8.

Bottom line, if you are an IT Tech, or expert of any kind, do not try to fast-talk a federal judge with “computerese” and specious theories.  It may fool your boss, and many attorneys, and make you look good for a while, but it will not work in court, and could get you in serious trouble.  If mistakes were made, then admit it. Don’t try to cover it up with technical jargon.  The best advice is to tell the truth and play it straight.  Also, be careful what you say in an affidavit or expert report.  You never know when you may be required to testify at trial to back it up.  You will then be subject to cross-examination, sometimes by a very skilled and knowledgeable attorney, and contradicted by a well-credentialed expert. Finally, from the attorney’s perspective, it is rarely a good idea to do what defense counsel did in this case, and go into an evidentiary hearing on complex IT issues without an impartial outside expert.  It is too dangerous to rely solely on the client’s own IT staff.  As this Jane Doe opinion shows, they can zero out your case real fast.

GhostSurfer Wipe Out Leads to Jail Order Sanction in Bankruptcy Court

A debtor in a bankruptcy was recently caught destroying evidence using a popular software program called GhostSurf.  United States v. Krause (In re Krause), 2007 WL 1597937, 2007 Bankr. LEXIS 1937 (Bankr. D. Kan. June 4, 2007).  The debtor was an attorney representing himself.  The case proves the old adage that “a lawyer who represents himself has a fool for a client.”  The lawyer used GhostSurf to try to wipe all incriminating evidence from his computers before producing them to the government.  His GhostSurf wipeout failed, and he was ordered to jail as a result.

The lawyer-debtor owed over three million dollars in back taxes and claimed poverty. He resisted e-discovery at first, but was eventually ordered to produce his computers for imaging and inspection by the government’s forensic experts. Immediately after the court order, the lawyer installed GhostSurf on his computers and used it to super-delete thousands of files from his hard drives. This software is designed to allow anonymous internet surfing.  It includes an application called “Tracks Cleaner,” which tracks and cleans files in all applications. It is similar in operation to another well known file shredding  program called “Evidence Eliminator” discussed in Kucala Enterprises, Ltd. v. Auto Wax Co., Inc., 2003 WL 21230605 (N.D. Ill. 2003).

The bankruptcy court’s description of GhostSurf’s “super-deletion” functions is very informative.  It makes it easier to understand the differences between: (1) simple deletion, where you just delete a file or email one time; (2) ”double deletion” where you delete a file, and then also empty the trash; and, (3) ”super-deletion”, the method employed by GhostSurfer and other software like it.  Super-deletion is designed to meet Department of Defense specifications for total file shredding, beyond the reach of forensic experts.  The different types of file deletions and data remanence can be very confusing. The following excerpt from Judge Nugent’s 30-page opinion sheds some needed light on the subject:

GhostSurf is designed to wipe or eradicate data and files as part of its protective and security functions.  . . . in such a way that the data is actually overwritten, precluding the ability to recover or restore the files and data. Both experts agreed that when a user “deletes” files from a hard drive, the data remains intact. The act of deletion merely eliminates the “pointer” that allows the computer to locate the data on the hard drive. By using data recovery software, that data may be extracted (as, indeed, some has been in this case). An additional step is necessary to eradicate this data entirely. GhostSurf performs this function by overwriting the file with a new file that contains no bytes of data and is named in a manner inconsistent with Windows operating system naming conventions. Rather than simply eliminating the pointer to the data, the actual recording of the data on the hard disk is erased (like taping over an existing tape recording).

Deleted e-mail leaves a different set of tracks. When a user “deletes” an e-mail in Outlook Express, the “fields” are deleted and sent to the trash or recycle bin. What remains on the hard drive are the HTML internet codes that define the fields, font, graphics, etc. of each message. What also remains is the actual e-mail message. When the trash bin is emptied, the matter itself is deleted. Because e-mail files are internet files, each time they are accessed, a temporary internet file (”temp file”) is created. Thus, even though the e-mail itself is deleted, the temp file remains on the hard drive, unless it is wiped.  . . .

Taylor testified that GhostSurf wipes files by searching the hard drive for files that Windows “no longer knows about” because they have been previously deleted, and writing data over those locations with random data to obscure it from undeleting. Once the files are overwritten in this fashion, an undelete utility cannot recover them.  . . .  According to the GhostSurf User’s Manual, the application may be set to erase files using different strength algorithms. If the weaker algorithms are used, the manual suggests “nearly all” of the targeted files will be erased. In short, GhostSurf is a very powerful tool that Krause could easily have used to purge files and data from his computers before turning them over to the Trustee.

Id. at *5, *7

The popular file wiping program did its job effectively.  Following Department of Defense computer file erasure protocols, it erased the files multiple times, rewrote the affected hard drive space with zeros, and set up fake file names.  Bottom line, there was no way to recover these files.  They were super-erased, and the forensic experts could not restore them.

But, the lawyer slipped up in at least two ways, and his scheme to destroy eviden