New Case where Police Use Hash to Catch a Perp and My Favored Truncated Hash Labeling System to ID the Evidence

Part of my discipline as an e-discovery specialist is to try to read (or at least skim) every published opinion on the subject. Lots of attorneys specializing in this area do that. But there is one other type of case I also read, every opinion that uses the word “hash.” No, I do not need help from Narcotics or Overeaters Anonymous. The kind of hash I am addicted to is purely algorithmic. This hash comes in many flavors, but the best known, and the ones usually employed in e-discovery, are called MD5 hash, SHA-1 hash, or the latest and greatest, SHA-2 hash. 

As I explain in my blog Hash page, hash is the mathematical foundation of e-discovery and the most powerful tool of any forensic investigator. It reveals the unique mathematical fingerprint of every computer file that allows for perfect identification and authentication of electronic evidence.  I became fascinated with the powers of hash a few years ago, and ended up writing a lengthy law review article on the subject. HASH: The New Bates Stamp, 12 Journal of Technology Law & Policy 1 (June 2007). A few months ago I wrote a blog on the article called The Days of the Bates Stamp Are Numbered, talking about some of the more recent developments in this area of the law, especially the shift from Tiffing and linear flat file Bates stamping to native file hash marking.

In the process of researching the original law review article, I am pretty sure I read every legal opinion and legal article ever written that  mentions hash. I also read a few scientific and cryptological articles as well, most of which I did not really understand. Having put that much time and effort into the subject, I try to keep up by reading every new legal opinion or article mentioning hash. That is why I have a standing search for all cases using the term, and automatically receive a copy of them by email as soon as they are published. I can be in the middle of dinner and my blackberry will buzz alerting me of a new hash case. Lest you think that’s a tad weird, I am willing to bet that there are a few other hash enthusiasts out there, Craig Ball comes to mind, who do the same thing. (See Craig Ball’s excellent article “In Praise of Hash” at pg. 52.)

Hash and Child Pornography

Most of the new hash cases I see have nothing to do with e-discovery per se. Instead, they are usually criminal law cases, typically cases involving one of the most disgusting of crimes, child pornography. Police have been using hash to catch perps in this area for years. Hash is an effective tool for this because it allows police to know if certain child pornography is located on a computer, usually videos or still photos, by looking to see if the hash values for these files are present. That is a bit of an over-simplification, but suffice it to say that there are lists of hash values that are known to be associated with computer files which are unquestionably child pornography. New York Attorney General Andrew Cuomo explained the process in a press release in June 2008 announcing a deal with major Internet providers to block major sources of child pornography: 

As part of the undercover investigation, the Attorney General’s office developed a new system for identifying online content that contains child pornography.  Every online picture has a unique “Hash Value” that, once identified and collected, can be used to digitally match the same image anywhere else it is distributed.  By building a library of the Hash Values for images identified as being child pornography, the Attorney General’s investigators were able to filter through tens of thousands of online files at a time, speedily identifying which Internet Service Providers were providing access to child pornography images.

U.S. v. Warren

I recently received a new hash case alert from a district court in Missouri. U.S. v. Warren, 2008 WL 3010156 (E.D.Mo. July 24, 2008). A quick review showed it was yet another child porn case, so I did not think much about it. I just added it to my reading list for more careful study later, just in case there might be something special about it. When I got around to reading Warren yesterday, I was very pleasantly surprised, as this was indeed a special case.

Warren is a case considering and rejecting a motion to suppress evidence, namely computer video files of underage teens having sex. The motion to suppress was based on a series of hyper-technical challenges to the affidavit which the St. Louis police submitted to the judge to receive a search warrant of defendant’s computer. The affidavit explained how the police had searched the Internet for files “whose digital SHA-1 value was identical to that of a file known to contain child pornography.” They found a computer with an Internet Protocol address of 70 … 167 offering to share one such known file, and then subpoenaed AT&T to get the physical address of the subscriber with that IP address. The computer was located in Affton, Missouri.

The police detective’s affidavit explained how the hash values and offer to upload established “that a computer in Missouri was ‘offering to participate in the distribution of known child pornography.’” Based on this affidavit, the judge found probable cause to issue the search warrant of the computers located in Warren’s home. The police then went to his home, found no one there, forced entry, and seized his computer. Warren himself later came along, and, foolishly enough, voluntarily came to the police station, waived his right to counsel several times, and spoke at length to the police. The opinion includes extensive excerpts of the taped interview, which Warren later argued was made in violation of his right to legal counsel.

The defendant’s technical search warrant objections forced the court to delve into many of the characteristics and evidentiary properties of hash. For that reason alone, the case is useful to any practitioner trying to better understand the subject. But what is really special about the case, at least for me, is the system of hash file identification used by the court to identify the offending video tape at issue in this case. That video computer file was the key piece of evidence, the “smoking gun.”

Six-Place Hash Truncation Naming Protocol

The opinion by Magistrate Judge David D. Noce in Warren is unusual and special because it is the first case to use the truncated hash value labeling system I proposed in HASH: The New Bates Stamp. My article was not mentioned, and apparently Judge Noce was not aware of it. He used the six-place hash truncation system I proposed in my article because it was, in his words, “convenient” to do so, and because the detectives had used that system in their affidavits and testimony. I doubt the police detectives had read my law review article either, which makes their use of the abbreviation system all the more important. It shows that it is a natural and reasonable thing to do, although this is the first time it has been utilized or mentioned in a legal opinion.

So what is the six-place hash truncation system which I proposed that these Missouri officials are now in fact using? Before I can answer that, I have to go into a little more depth about hash and Bates stamps. HASH: The New Bates Stamp not only explains hash and its importance to e-discovery, it also argues for the legal profession and e-discovery industry to adopt a new type of electronic document naming protocol that uses hash values, instead of sequential numbering, to identify electronic evidence. I argue that the time has come for the legal profession to abandon Nineteenth Century Bates stamp paper mentality, and adopt Twenty-First Century ESI hash mentality. I proposed that sequential Bates stamps be replaced by non-linear, intrinsic hash values.

The hash values would not only identify ESI, they would authenticate it too, something the lowly Bates stamp could never do.  But the problem with using hash values to identify ESI, instead of Bates stamps, is that hash values are too long and awkward for the human mind. Here is what a typical forty place hexadecimal SHA-1 hash value looks like: 2B37BC6257556E954F90755DDE5DB8CDA8D76619.

Police detectives, lawyers and judges cannot go around describing computer files used as evidence with such long alphanumerics. It is too cumbersome a name to replace the Bates stamp. So my common sense proposal, which Judge Noce in Warren calls “convenient,” is to only use the first and last three places of the hash value, instead of all forty. So the hash value above becomes the much more manageable 2B3 … 619. That truncated hash value becomes a pretty good document name, and, in my opinion and that of many others, should replace the arbitrary Bates stamp.

Turns out that the detectives in Missouri were already following this six-place truncation protocol at the time my article was published in June 2007. Perhaps they and other law enforcement agencies have been using this system for years. I do not know for sure, although I doubt it has been a widespread practice. I have talked to many e-discovery forensic experts about the hash naming proposal over the past two years. Many of these experts did police work before going into e-discovery, and none ever mentioned having done this before. Also, it certainly does not appear in the legal literature on the subject, that is, until U.S. v. Warren.

Hexadecimal Values v. Base32 Number System

At first, I was disappointed to see that Judge Noce’s introduction of the truncated hash value naming protocol was flawed with two obvious technical errors. See if you can catch them:

The search turned up a list of files, including one with a 32-character alpha-numeric SHA1 designation of “H4V … UTI.” Fn4

FN4 - For convenience, in this opinion the SHA1 value set out in full in the search warrant affidavit will be referred to as “H4V … UTI.” The affidavit defined the term “SHA1” (also known as “SHA-1”) as being a mathematical algorithm that uses the Secure Hash Algorithm (SHA), developed by the National Institute of Standards and Technology (NIST), along with the National Security Agency (NSA) . . . Basically the SHA1 is an algorithm for computing a condensed representation of a message or data file like a fingerprint.

Warren at *1.

First of all, the SHA-1 hash generates a 40-character hexadecimal string, not 32-character. The other kind of hash, MD5 hash, is the one that uses a 32 character string, not SHA-1. For this reason, my first reaction was that the Judge, or police, mixed up the two different types of hash, and meant to say 40 characters, not 32.

But then there seemed to be yet another, even bigger mistake. The letters H V U T and I should not have been in the hash value name. The values generated in e-discovery work to represent SHA-1 and MD5 hash are always hexadecimal. That is a numerical system with a base of 16. This is typically represented by the numbers 0–9 for the first ten values, and A, B, C, D, E, and F to represent the last six, for a total of sixteen. In other words, a hexadecimal value does not employ any letters after F. Yet, the so called SHA-1 alphanumeric stated in the Warren opinion uses the letters H, U, T and I: “H4V … UTI.”

I thought the police or Judge Noce must have messed things up, but I also seemed to remember reading somewhere that were other ways to express hash values, and anyway, I am always very careful before I tell a judge that he or she is wrong. So doing a little online research, I learned that there are indeed other ways to display hash values using different binary based number systems, typically the 32 base or 64 base number systems. Base32 is defined in IETF RFC 3548, as using the characters A-Z and 2-7. While Base64 is defined in IETF PEM RFC 1421 as using the characters A-Z, a-z, 0-9, / and +.

My Online Investigation of Base32 Hash Math
Led to a Shocking Discovery

Coming back to the Warren opinion, the hash values “H4V … UTI” are not hexadecimal, but they could be either Base 32 or Base 64. At this point, I did a little more online research about Base32 hash, and quickly found that there are many websites where you can locate music and videos to download based on their hash values. Almost right away, by simply using Google, I located a site where you can find media to download based upon their SHA1 Base32 value. It then took less than a minute to find the web page where the Base32 SHA-1 hash values were listed that began with “H4V.” That is how all of the media on the site was listed, in numerical order based upon the first three numbers of their Base32 hash values.

There were 83 entires on the webpage whose hash values began with H4V. The site included listings of music and videos ranging from Beethoven’s Symphony No. 9 to a video of Lee Trevano’s Golf Instruction. One video listing which was 11.1 MB in size had a disturbing title that suggested it could contain the kind of porn referenced in Warren. It was dated May 29, 2003. I clicked on its hash value button and saw that the full SHA-1 hash value for this video was H4VIBLSKAZ477WRTKH7IURE6NXEDCUTI.

When I saw that hash value, it shook me up. The first and last three values exactly matched the hash described in Warren: H4V … UTI. My academic investigation of the mathematical properties of hash had led me right to the smoking gun in Warren! I knew from my article, and the research of Bill Speros described in footnote 168, that this match of the first and last three values meant there was a 98.6% probability that this was the exact same file referenced in Warren.  Mr. Warren was charged with a felony for distributing this same video. I think it is a crime to even have it on your computer.

I do not know for sure if it is the same file, since the Warren opinion nowhere states the full hash value, but in view of the description of this video, it is just too much of a coincidence for it not to be.  It was astonishing on many levels to see just how quickly you can find a file like this on the Internet, simply by knowing the first three hash numbers. 

It is probably not possible to actually download or view the file from this website. I do not really know for sure, since that would involve clicking on this file, which I was not about to do. But when I clicked on the link for Beethoven’s Symphony No. 9, a piece of media which I do not find morally reprehensible, it took me to another web page. This page had links to other computers where you may in fact have been able to download Beethoven’s music. (I did not try, recognizing that might be a copyright violation.) At that point, the referring website included a statement that it “ONLY HAS INFO ABOUT FILES, AND DOES NOT OFFER ANY FILES FOR DOWNLOAD.” Still, if any law enforcement agency wants to contact me for the full website address, including Cuomo’s group, I would be happy to provide it. It is really very easy to find, and so I assume the proper authorities are already well aware of this site and its hash values, or lack thereof. I am certainly no police officer, and even if I was, I would not have the stomach for this kind of investigative work. Reading the email of parties in civil suits is about as horrid as I can handle.

Judge Noce Was Right

This little investigation proved to me that Judge Noce and the St. Louis police were correct. There is a SHA-1 hash that has 32 places, not 40, and it can use the whole alphabet, not just A-F.

The hash value H4V … UTI is indeed a correct first and last place truncation of a full SHA-1 hash value. But it is a SHA-1 hash that is expressed in Base32, not hexadecimal. Although the hash values used in e-discovery are almost always hexadecimal, the hash values used in “Peer-to-Peer” websites include a variety of different numerical systems, frequently including the Base32 system.

In addition, in my brief investigation of the P2P webs, I learned that countless P2P type websites now commonly use the first three places of hash values as a convenient shorthand naming system. For all I know, the “perps” may also. As Judge Noce says, it is the convenient thing to do. So when will the e-discovery vendors start doing so too?

More “Must Read” 2008 Cases - Part Two

This is Part Two of a three part series reviewing, in alphabetical order, some thirty 2008 e-discovery cases. These are particularly interesting and noteworthy cases, but ones not previously featured in this blog. Part One covered ten cases A-J. Here in Part Two, seven more cases are written up, using some 4,150 words, but I only get through two more letters in the alphabet, H-I. That is because there are so many good “In Re” type cases, which are usually complex cases consolidated in a single court. These cases frequently involve important e-discovery issues, as you will soon see.

The first case here is the lone “H” case, Henry v. Quicken Loans, Inc., 2008 WL 474127 (E.D. Mich. Feb. 15, 2008). Henry involves search protocols and a questionable decision to change the agreed-upon, and court-ordered, procedures without telling the other side, or seeking permission from the judge. The not-so-subtle message of Henry is the need for communication and openness in e-discovery, especially when attempting a complex search of a large volume of ESI. The old ways of extreme adversarial discovery conduct do not work anymore in the new world of complex e-discovery. You may think you are helping your client, but, as this first case shows, you are not. More often than not you just end up making a bad situation worse. 

Henry v. Quicken Loans, Inc., 2008 WL 474127 (E.D. Mich. Feb. 15, 2008).

This case was brought by 422 plaintiffs who worked for Quicken Loans as “loan consultants.” They sued under the Fair Labor Standards Act for time and a half overtime pay for all work over 40 hours per week. Plaintiffs’ requested production of emails which they and 32 managers had written during a three month period in 2004. Apparently these emails were only available on backup tapes, and so Defendant objected.

Plaintiffs filed a motion to compel production and establish a protocol for production and clawback type privilege review. The motion was granted and the court compelled production and “a protocol was ordered that was intended to balance the concerns and needs of both sides at what was hoped to be manageable costs.” Id. at *1. The protocol called for Plaintiffs’ computer expert to retrieve email from the tapes under Defendant’s direction, but using search and screening terms agreed to by the parties. Plaintiff’s counsel agreed to pay for the expert’s expense, but “expressed a desire to limit the costs to no more than what was needed and not to be giving defense counsel a “carte blanche” to run up the costs of the screening procedure at Plaintiffs’ expense.” Id. at *2.

Mid-way through the screening process defense counsel directed the computer expert to start using different keywords to screen emails and attachments. This is difficult to understand. If defense counsel found that the search terms the parties had agreed to were not working well, which is certainly possible, if not likely, and felt it was necessary to change the terms, then why not speak with Plaintiffs’ counsel about it, or the court? (Later in the opinion the court stated that it would have agreed with some of the defense changes, but not all, if it had been asked in advance.) This was, after all, the Plaintiff’s expert, and so, defense counsel could not possibly have thought this “departure” from the agreement would pass undetected. So, why did defense counsel make a unilateral decision to change the search terms? By all appearances, it looks like an intentional violation of the parties agreement and court order? Monday morning quarterbacking is always easy, but still, this one looks like a no-brainer. So why did this happen?

Apparently defense counsel did think that this “little mid-course correction” could go by undetected. Yes, it was originally Plaintiffs’ expert, but the court order made clear that the expert would work under the “direction and control” of Defense counsel and would maintain confidentiality. Further, to underscore this point, and develop and extend the order a bit, Defense counsel told the expert that he “would be acting under the ‘direction and control’ of defense counsel and was not to have any contact with Plaintiffs’ counsel regarding his work.” Id. Apparently defense counsel figured they had set it up so that they could make these changes in search protocols without being detected, and so limit the production to Plaintiffs. Alternatively, they figured that if they were caught, there would not be significant adverse consequences. Wrong on both counts.

Here is how they were found out. The expert presented his bill to Plaintiffs’ counsel for payment as the court order provided. It is interesting to note that the bill was originally for $91,191.35, but was subsequently reduced to $79,965.01 due to “inadvertent double billing for certain work.” Id. at *3. (Seems odd - over twelve thousand dollars in inadvertent double billing?)  As soon as Plaintiffs’ counsel got the hefty bill, he sent a copy to Defense counsel, and advised them of his intent to depose the expert “regarding his bill and the protocol he followed.” Id. It seems to me that this should have been expected. But apparently not.

On the day of the noticed deposition, Defense counsel filed a motion for protective order to try to prevent the deposition. This delayed the deposition by several weeks, but after the motion was denied, the deposition eventually went forward. Of course, that is when the truth came out. The expert told all about how Defendant’s attorneys forced him to use new terms and procedures. Not surprisingly, the result of the new terms and procedures was an under-inclusive production of emails. The expert guessed at his deposition that “tens of thousands” of emails were not produced as a result of the change.

Again, as would be expected, Plaintiffs’ counsel responded with motions to the court. In my view, the motions were pretty mild considering the events described in the opinion. I would have expected a motion for sanctions. Instead, Plaintiffs counsel sought only to have the expert costs shifted, asking for the Defendant to pay most, but not all, of the expert’s bill, not Plaintiffs. Here is the court’s explanation:

*5 While there is no suggestion that defense counsel was acting unethically in serving his client’s interest, defense counsel’s actions exceeded the scope of the “direction and control” powers this Court vested in him in his unilateral and unauthorized modification of the “specific parameters” of the July 10, 2007, letter.

If there was a belief before or after the first screening run that there was some ambiguity in the meaning of screening for the names of identified company legal personnel, where those names were listed as a usual combined first and last name, defense counsel should have aired that with Plaintiffs’ counsel. If agreement could not be reached, Plaintiffs’ or Defendants’ counsel could have raised the matter with the Court. Had the issue of screening separately for first names of the 14 attorneys been brought to the Court’s attention, the initial interpretation and recommendation of Mr. Lanterman [the expert] of using both names together would have been adopted, or if separation was to occur, the screening would be done using the last name only of the attorneys. Instead of this, defense counsel unilaterally and secretly resolved any ambiguity, if such exists, in his client’s favor and against the advise of the forensic computer screening expert.FN5 It appears that as a result of this all e-mails screened subsequent to the initial screening were corrupted as being under-inclusive. Thus, it is determined that it is not a reasonable expense to impose the costs of subsequent screening runs on Plaintiffs where the corruption was caused by the acts and omissions of defense counsel, and could have been avoided.

Plaintiffs’ motion was granted and Defendant was ordered to pay the expert fees. Plaintiffs’ have another motion pending to have the court find a waiver of privilege (which may explain the seemingly mild response here to the hidden change). This opinion did not explain the waiver issues, but did state that depending on how it rules on this privilege motion, Defendant may be required to produce all of the emails it screened from the unilateral revision, and then some. The decision not to call Plaintiffs’ counsel, or the court, and instead unilaterally change the search procedure, could indeed have very harsh consequences.

In Re Flash Memory Antitrust Litigation, 2008 WL 1831668 (N.D. Cal. Apr. 22, 2008).

In this case, the court entered a preservation order reminding parties of their duty to preserve all potentially relevant evidence. It appears if as this was done sua sponte as there is no reference to a motion or stipulation. The parties were directed to take reasonable steps to ensure that potentially relevant records are preserved and to notify all parties and non-parties of their duty to preserve.

This is a one page order that simply reminds the parties to do what the law already requires. No explanation was provided for why the court felt it necessary to help the parties with their memory in this way, but I cannot help but notice the irony in view of the title of this case. The Flash Memory opinion concludes with the following:

Until the parties reach agreements on a preservation plan or the Court orders otherwise, each party shall take reasonable steps to preserve all documents, data, and tangible things containing information potentially relevant to the subject mater of this litigation. In addition, counsel shall exercise all reasonable efforts to identify and notify parties and non-parties of their duties, including employees of corporate or institutional parties, to the extent required by the Federal Rules of Civil Procedure.

In Re Honza, 2007 WL 4591917 (Tex. App. Dec. 28, 2007).

Technically this may not be a 2008 case, but it is real close. It is a state appeals court opinion involving real estate litigation. The appellate court upheld a trial court order requiring a forensic examination of Defendants’ computers so that outcome determinative metadata could be discovered.

The main issue in the underlying case is whether Defendants had altered a partial assignment of a real estate contract after the parties finalized the agreement. The trial court ordered a computer expert to create a mirror image of the computer hard drives in Defendants’ office in order to locate all drafts of that assignment, including deleted files. Plaintiffs wanted the documents in their native format with the metadata intact so that they could determine when the drafts were modified. The appeals court found that under these circumstances, the order was not overly broad and was properly crafted to protect disclosure of privileged and confidential information. 

Here is the court’s explanation of why acquisition of the metadata in this case was critical:

A & W seeks the metadata from the Honzas’ hard drives because it wants to identify the points in time when the partial assignment draft was modified in relation to the diary entry. This goes to the issue of whether the Honzas altered the partial assignment after the parties concluded their agreement but before the document was presented for execution.

There is no explanation as to why the Defendants did not simply produce the native files with all metadata. Apparently, the defendants simply refused to do so thus triggering the rationale for the forensic examination. This fact also makes it easy to distinguish this case, because, in most circumstances, the producing party would simply produce the metadata, at least as a fall back position. In a case like this, where only two documents were involved, and the metadata could be outcome determinative, the metadata was obviously relevant and should have been produced voluntarily.

The Texas appeals court opinion is noteworthy for its good review of some of the many state and federal cases addressing the issue of forced forensic examination of a producing party’s computers. This was an issue of first impression in Texas, but has been considered by many other state courts, and dozens of district courts. The opinion also contains a detailed list of the procedures to be followed during the exam to protect the rights of the examined party. It is a good decision to know about, especially in state court cases.

In Re Seroquel Products Liability Litigation, 244 F.R.D. 650 (M.D. Fla. Aug. 21, 2007).

Ok, this is not a 2008 case either, but it is an important case that has a lot to say about search and the need to cooperate in e-discovery. I should have written about Seroquel previously; so, better late than never. This is a well-written decision by Orlando Magistrate Judge David Baker. Judge Baker is not yet one of the “Judicial Rock Stars” of e-discovery, but he has long experience with computers and a good understanding of e-discovery, and could easily join the “e-disco” circuit if he wanted to.

This is a multi-district products liability action. It was filed by 6,500 Plaintiffs who took the drug Seroquel, an anti-psychotic medication that is purported to cause diabetes and other illnesses. I can only imagine the kind of client relations problems the Plaintiffs’ lawyers must have. This opinion addresses Plaintiffs’ motion for sanctions against the Defendant, the drug’s manufacturer, AstraZeneca, due to its failure to timely or properly comply with multiple discovery requests. After a raucous hearing, Judge Baker granted the motion and imposed the sanctions, even though there was no showing of willfulness or bad faith.

There was, however, proof of massive discovery negligence bordering on bad faith that caused unnecessary expenses and delays. Judge Baker characterized Defendant’s response to discovery as “purposeful sluggishness.” Defendant eventually produced 25 million pages of ESI, but of this, over 10 million pages were inaccessible, unsearchable, and unusable. The was at the heart of the sanctions motion. The flat TIFF images produced were not searchable because they were not accompanied with any metadata or other load file. Further, the production included multiple giant “TIFF” images, some over 20,000 pages long, with no page breaks. When plaintiff’s counsel suggested the inclusion of page breaks, defense counsel called it “ingenious,” a fact which Judge Baker pointedly noted in the sanctions opinion.

Judge Baker criticized both sides for their lack of professionalism and over-argumentative approach to electronic discovery. But, the Court found primary responsibility for the breakdown in communication and cooperation on the Defendant whose records were at issue. The following quotes from the opinion are of particular importance, and are starting to appear in other decisions nationwide, including the latest hot opinion on search issues by Judge Grimm, Victor Stanley.

While keyword searching is a recognized method to winnow relevant documents from large repositories, use of this technique must be a cooperative and informed process. Rather than working with Plaintiffs from the outset to reach agreement on appropriate and comprehensive search terms and methods, AZ undertook the task in secret.

Common sense dictates that sampling and other quality assurance techniques must be employed to meet requirements of completeness. If AZ took such steps, it has not identified or validated them.

One day all practitioners will understand and agree that using sampling and other quality assurance techniques in ESI search is just plain common sense. In the meantime, judges like David Baker, Paul Grimm, and many others stand ready to sanction clueless lawyers whose blundering, or adversarial based “purposeful sluggishness,” needlessly drives up the cost of litigation.

In Re Intel Corp. Microprocessor Antitrust Litigation, 2008 WL 2310288 (D. Del. June 4, 2008).

In this antitrust litigation, supposedly the largest in the country, the court found waiver by the defendant Intel of attorney-client privilege and work product protection. By the time this class action is concluded, all parties expect it will involve the largest production of ESI in history:

AMD, the Class Plaintiffs and Intel recognize that this litigation could be the “largest electronic production in history” resulting in Intel’s production of “somewhere in the neighborhood of a pile 137 miles high.” (citations omitted)

Intel discovered some lapses in its retention of potentially relevant documents and disclosed these lapses to the court and the many opposing parties, most notably AMD. This then forced Intel to start to look for these deleted emails on its backup tapes and other locations. It then discovered that some of the backup tapes had also not been preserved. This triggered a full investigation, including interviews by new outside counsel retained for that purpose, Weil Gotshal & Manges, LLC (”Weil”). In what any law firm would consider a dream assignment, Weil was retained to interview each of Intel’s 1,023 identified custodians “for the purpose of determining their e-mail preservation habits and their level of compliance with Intel’s litigation hold notices/instructions.” As part of these remedial efforts, Weil attorneys “spent many hours preparing roughly 400 pages of custodian-specific retention reports which drew upon thousands of pages of attorney notes, as well as other information.”

Plaintiffs sought the production of the Weil attorney notes underlying these retention reports in order to determine whether there was any deliberate destruction of relevant data. Intel opposed production, asserting attorney-client communication privilege and attorney work product privilege to all of the notes. A Special Master was appointed to determine if Intel had waived these privileges. The Special Master found waiver of attorney-client privilege because Intel had asserted to the Plaintiff, and the court, that the deleted information was due to human error and not intentional destruction. Intel waived work product protection when it agreed to produce detailed descriptions of the preservation issues of each custodian. This was waiver of “fact work product” only, and did not include “core work product,” a/k/a “opinion work product” which the decision notes is accorded near absolute protection from discovery. Id. at pg. *15. The court fully adopted the Report and Recommendation of the Special Master and overruled all of Intel’s objections.

The Court felt that waiver was required; otherwise, Intel would be “in the position of being able to use its sword to assert facts while at the same time shield AMD, the Class Plaintiffs and the Court from the accuracy of Intel’s assertion.” This would in effect force AMD and Class Plaintiff’s to either take Intel’s word for everything, or start all over and depose the Intel witnesses themselves. That would have defeated the whole purpose of the original discovery agreement under which the Weil investigation was commenced. As the Court explained:

In the context of the discovery phase of Intel’s preservation issues, Intel, by and through its attorneys, agreed to this form of discovery. Intel agreed to produce, “detailed written description[s] of the preservation issues affecting [every] Intel Custodian, including the nature, scope and duration of any preservation issue(s).” Intel could have left AMD and the Class Plaintiffs to their own devices, forcing them down the path of protracted world-wide preservation depositions. It did not. Rather it trumpeted its willingness to have AMD, the Class Plaintiffs and the Court informed as to fact work-product gathered and provided “a detailed written description of the information provided by each custodian [to Weil] during the interviews.”

CONCLUSION:

*17 The Special Master concludes, therefore, that Intel cannot now mask its agreed to discovery of custodian information by asserting the work-product privilege with respect to fact work-product which, in the Special Master’s view, lies at the heart of Intel’s position on its preservation issues.

So the troubles for Intel in this antitrust case continue. Not only will its mistakes in preservation cost Intel a small fortune to try to correct, but now it will also cost Intel some of its attorney-client secrecy protection. This could in turn lead to more sanction motions by AMD and other class plaintiffs, depending in no small part upon the findings and conclusions of Intel’s own lawyers. 

In Re Subponea Duces Tecum to AOL, 2008 WL 1956266 (April 18, 2008).

In this qui tam action against State Farm Insurance, the court considered a motion objecting to the magistrate judge’s order quashing State Farm’s subpoena of AOL. The filers of the qui tam action (called “relators”) argued that the subpoena to AOL violated the Electronic Communications Privacy Act. State Farm subpoenaed AOL for certain information relating to the email accounts of listed State Farm adjusters. The subpoena also requested all emails and electronic documents sent and received during a six-week period from one of the adjuster’s email accounts. The adjusters were not parties to the underlying litigation.

The court upheld the magistrate judge’s order for two reasons. First, the subpoena was over-broad due to the fact that the request for all emails and electronic documents during the six-week period was not limited to documents relevant to the subject matter of the litigation and it imposed an undue burden on the third party insurance adjusters. The court reasoned that the emails and electronic documents produced from the six-week period would likely contain privileged and personal information that was unrelated to the litigation. Second, the court held that:

[T]he plain language of the Privacy Act prohibits AOL from producing the Rigsbys’ e-mails, and the issuance of a civil discovery subpoena is not an exception to the provisions of the Privacy Act that would allow an internet service provider to disclose the communications at issue here.

This is an important ruling construing the Electronic Communications Privacy Act (”Privacy Act”). 18 U.S.C. §§ 2701-03 (2000). It applies only to civil discovery subpoenas of Internet service providers, and does not apply to civil trial subpoenas, nor to criminal subpoenas. Here is a segment of the Court’s reasoning behind this important ruling:

Protecting privacy interests in personal information stored in computerized systems, while also protecting the Government’s legitimate law enforcement needs, the Privacy Act creates a zone of privacy to protect internet subscribers from having their personal information wrongfully used and publicly disclosed by “unauthorized private parties,” S.REP. NO. 99-541, at 3 (1986), as reprinted in 1986 U.S.C.C.A.N. 3555, 3557.

*4 Applying the clear and unambiguous language of § 2702 to this case, AOL, a corporation that provides electronic communication services to the public, may not divulge the contents of the Rigsbys’ electronic communications to State Farm because the statutory language of the Privacy Act does not include an exception for the disclosure of electronic communications pursuant to civil discovery subpoenas.

[T]he Court holds that “unauthorized private parties” and governmental entities are prohibited from using Rule 45 civil discovery subpoenas to circumvent the Privacy Act’s protections.

This means that if you want to discover email you need to subpoena the subscriber, the person whose email account it is, not the provider. If you receive a court order compelling the individual subscriber to produce the email, then, and only then will the Internet provider cooperate.

In Re World Trade Center Disaster Site Litigation, 2008 WL 793578 (S.D.N.Y. Mar. 24, 2008).

This litigation involves approximately 10,000 cases relating to the respiratory injuries of workers exposed to toxic chemicals in the clean up and rescue efforts after the terrorist attacks of September 11, 2001. The court recognized the massive volume of e-discovery required to fairly resolve the many claims in this important case. For that reason, this opinion accepts and approves the Special Masters recommendation that Technology Concepts & Design, Inc. be hired to build, maintain, and operate a computer database to store “Core Discovery.”

The recommendation was made by the multiple Special Masters previously appointed in these consolidated cases. (By the way, the opinion notes that the Special Masters’ bills average $20,000 per month.) The court accepted the recommendation because, as the court put it, “it is imperative that a means be developed to process and organize the information on a consistent, reliable, and accessible basis.”

The Plaintiffs had opposed the hiring of the expert, and construction of the database, largely on the basis of the additional cost involved. Plaintiffs were keenly concerned with costs because all of the expenses are paid out of a capped recovery fund. The court rejected these arguments, claiming that a source of reliable information was indispensable to the efficient and inexpensive resolution of these 10,000 plus cases. The Court also explained its authority to appoint Special masters and experts to assist courts in complex discovery issues:

The court has ample power to appoint Special Masters, and to support their activities in manners that will create efficiencies and economies. Fed.R.Civ.P. 53; In re Peterson, 253 U.S. 300, 312-13 (1920) (”Courts have … inherent power to provide themselves with appropriate instruments required for the performance of their duties. This power includes authority to appoint persons unconnected with the court to aid judges in the performance of specific judicial duties” including Special Masters); Manual for Complex Litigation §11.52 (4th ed. 2004).

The use of Special Masters in this case is a good example of an emerging trend in e-discovery. Like most people in the field, I predict a significant increase in the use of Special Masters for e-discovery specific issues in the coming years. These issues are frequently too complex and time consuming for district court judges or magistrates to deal with on their own. For consistency, predictability, and quality of ruling, not to mention speed, more and more parties to complex litigation will voluntarily agree to use a special master with special skills and experience to resolve their e-discovery disputes. Even if they do not all agree, the judges may well force it upon them. As this opinion shows, they have ample legal authority to do that, especially in an extraordinary case like this.

 

More “Must Read” 2008 Cases - Part One in a Three Part Series

This is the first of a three-part blog, and a follow-up to an earlier essay, Online Reference and Thirty One More e-Discovery Cases. In this and the next two blogs, I will add thirty new case summaries and analysis in alphabetical order. This first blog covers the first ten cases, A-G, from the first half of 2008. It is already pretty long at 4,698 words, and one Monty Python video on spam, which is why I am breaking this up into three blogs of ten cases each. These are all cases that I consider “must read” e-discovery cases for 2008. Of course, if you want to “have a life” outside of e-discovery, you might just settle for reading these summaries, and then come back and read the full case later, if and when you need to.

These 30 new reviews are in addition to the twelve lengthy articles on 2008 cases that I have already posted:

1. Qualcomm v. Broadcom.
2. Johnson v. Wells Fargo Home Mortgage, Inc.
3. R & R Sails, Inc., d/b/a Hobie Cat Co. v. Insurance Company of the State of Pennsylvania.
4. Victor Stanley, Inc. v. Creative Pipe, Inc.
5. Texas v. City of Frisco.
6. Petcou v. C.H. Robinson Worldwide, Inc.
7. Bro-Tech Corp. v. Thermax, Inc.
8. Victory Intern. (USA) Inc. v. Perry Ellis Intern., Inc.
9. ICU Medical, Inc. v. RyMed Technologies, Inc.
10. Diabetes Centers of America, Inc. v. Healthpia America, Inc.
11. United States v. O’Keefe.
12. D’Onofrio v. SFX Sports Group, Inc.

The above twelve cases are, in my view, the most important decisions so far in 2008, but the 30 others I will review today, and over the next two weeks, are no slouches either. The truth is, there are tons of good e-discovery cases this year, including many more that did not quite make the cut here for one reason or another.

These new cases will help make the e-Discovery Team blog more useful as an online reference tool for students, fans, and practitioners of e-discovery. In the future, just use the Search button on the right to help you find or remember the e-discovery cases you need. In the meantime, reading these short reviews should help keep you up to date. As usual, unlike the excellent K&L Gates Blog, I interject my own opinions and analysis into most of these reviews. Again, these are my views, not that of my firm or clients. I do this with the hope that some analysis may help in your own processing, or at least keep the study of the law a tad more interesting, and maybe even sometimes amusing. After all, learning should be fun, especially for cutting edge stuff like this.

Special thanks to Catherine Jackson, an Editor of Florida Law Review, and Summer Associate at Akerman, for her fine help on this project.

Autotech Techs. Ltd. P’ship v. Automationdirect.com, Inc., 248 F.R.D. 556 (N.D. Ill. 2008). In a trademark infringement case, Defendant filed a motion to compel Plaintiff to produce a document titled EZTouch File Structure in native format with all metadata. Plaintiff argued that it had already produced the document electronically in PDF format and in paper format. Additionally, Plaintiff provided a “Document Modification History” representing a chronological list of all changes made since the document was created. The court denied Defendant’s request because Defendant had neither specifically asked for native format in its original document requests nor mentioned native format in any of its earlier motions. The court reasoned that “[Defendant] was the master of its production requests; it must be satisfied with what it asked for.”

The authority the opinion cites at page *560 to support this holding includes three cases, D’Onofrio v. SFX Sports Group, Inc., 247 F.R.D. 43, 48 (D.D.C.2008); Wyeth v. Impax Labs., Inc., No. Civ. A. 06-222-JJF, 248 F.R.D. 169, 170-72, 2006 WL 3091331, at *1-2 (D.Del.2006); Treppel v. Biovail Corp., 233 F.R.D. 363, 374 (S.D.N.Y.2006), and then my book:

See also Ralph C. Losey, E-Discovery, Current Trends and Cases 158-59 (2007) (summarizing recent cases as amounting to a “lesson … that in order to obtain metadata you may need, you should specifically ask for it to begin with”).

It would have been a must read case anyway, honest.

Baker v. Gerould, 2008 WL 850236 (W.D.N.Y. Mar. 27, 2008). Plaintiff, a state employee, brought an action against his employer for failing to promote him based on the fact that he had exercised his constitutional rights. Plaintiff filed a motion to compel the production of emails. Defendants produced some responsive emails, but Plaintiff renewed the motion to compel based on inadequate production. After oral argument on the motion, the court ordered Defendants to submit an affidavit delineating the steps they took to locate the emails.

The Defendants filed an affidavit, but it did not describe the steps taken to find the emails as the court had requested. Instead, the affidavit outlined the necessary steps to restore deleted data from backup sources. Plaintiff argued that the emails should be produced even if Defendants had to restore deleted data.

The court examined the elements of Rule 26(b)(2)(B) for production of inaccessible ESI, such as deleted emails, and then made the interesting holding that the rule should not apply unless and until you first try and obtain the ESI from all accessible sources. In this case, that would include PST email archive folders that employees may have made on their personal computers. The existence of such files is unknown, which is one reason the affidavit on search was ordered to begin with. If the missing emails were located in these easy to access PST files, there would be no need to consider searching deleted space and thus no need to follow the analysis of Rule 26(b)(2)(B). In the words of the court explaining the rule:

Under this framework, a court does not reach the two-fold question of whether inaccessible sources of electronically stored information should be searched and, if so, which party should bear the associated costs unless it is first satisfied that the request seeks relevant information that is not available from accessible sources. . . . This is because relevant considerations in determining whether to order a search of inaccessible sources include “the quantity of information available from other and more easily accessed sources” and “the likelihood of finding relevant information that seems likely to have existed but is no longer available on more easily accessed sources.” Id. See also Zubulake v. UBS Warburg LLC, 217 F.R.D. at 323 (one of the two most important considerations is “the availability of such information from other sources”).

Since the defendant did not provide the court with the affidavit it requested to allow the court to determine whether accessible sources of the email might exist, such as the mentioned PST files, the court held that it still did not have enough information to determine whether it should force the government defendants to search through their deleted files or not. The court was obviously persuaded that such an effort would be expensive and time consuming, and seemed completely forgiving of the fact that the state defendants had previously ignored her prior request for information by affidavit. The court denied plaintiff’s motion to compel, but without prejudice. The defendants were ordered to submit to depositions by the plaintiff as to their purported efforts to locate email. After such discovery, which the court assumes will provide information on the availability, or not, or accessible email, the plaintiff can return to the court for appropriate relief

City of Seattle v. Prof’l Basketball Club, LLC , 2008 WL 539809 (W.D. Wash. Feb. 25, 2008). In a case involving a dispute over a lease, plaintiff City of Seattle requested defendant Professional Basketball Club (the Seattle “Sonics”) to search and produce responsive emails of six of its eight owners. The Sonics had already produced about 150,000 emails from two of its owners, Clay Bennett, and Aubrey McClendon. (Who knew professional basketball management types wrote so many emails!) The Sonics objected to the additional requests because such a search would “increase the universe exponentially” and would generally produce irrelevant documents. Still, I have got to believe that the Sonics email is a lot more interesting to read, and certainly more colorful, than the email in your typical landlord/tenant case. That is especially true in this case where these new team owners fought a huge public battle, of which this case is a part, to move the team from Seattle to Oklahoma City.

The City apparently really wanted to see the email of the rest of the Sonics owners too, so it moved to compel production. The court granted the motion because the Sonics had control over the owners’ emails under Federal Rule of Civil Procedure 34(a), the emails were likely relevant to the claims, and there were no specific objections as required under Federal Rule of Civil Procedure 26(b)(2)(B). The court ordered the production of the emails because the Sonics had not adequately explained why the production was unduly burdensome. In the words of the court:

In opposing discovery on the grounds of overbreadth, a party has the burden “to provide sufficient detail in terms of time, money and procedure required to produce the requested documents.” Super Film, Inc. v. UCB Films, Inc., 219 F.R.D. 649, 651 (D.Kan.2004) (citation omitted). A “court must be able to ascertain what is being objected to. As such, unless it is obvious from the wording of the request itself that it is overbroad, vague, ambiguous or unduly burdensome, an objection simply stating so is not sufficiently specific.” Boeing Co. v. Agric. Ins. Co., 2007 U.S. Dist. LEXIS 90957, *8 (W.D. Wash. Dec. 11, 2007). A claim that answering discovery will require the objecting party to expend considerable time and effort to obtain the requested information is an insufficient factual basis for sustaining an objection. Roesberg v. Johns-Manville Corp., 85 F.R.D. 292, 296 (D.C.Pa.1980).

Here, PBC has not explained why producing the emails at issue would be unnecessarily burdensome, but merely states that producing such emails “would increase the email universe exponentially[.]” (Dkt. No. 14, Ex. C). PBC also states in its moving papers that the emails add “nothing to the case except mountains of work for no return.”(Dkt. No. 14 at 7.) But a bald assertion that discovery will be burdensome is insufficient in light of Fed.R.Civ.P. 26(b) (2)(B). The Court is not permitted to presume the potential burdensome effects upon a party. The parties have already agreed upon a group of search terms that PBC previously used to search Messrs. Bennett and McClendon’s emails and the Court assumes those terms may be used again to make further searches efficient.

The lesson of this case is that you cannot rely upon general objections buttressed by big numbers alone to persuade a court to protect you from burdensome discovery. You have to spell it out and provide details as to why it is so burdensome. Otherwise, even if you are the Sonics in your home court in Seattle, you may not be heard. And if you are new owners trying to take the team away, it might not matter anyway, but you should at least try

Clearone Communications v. Chiang, 2008 WL 704228 (D. Utah Mar. 10, 2008). Plaintiff filed a motion for sanctions in this case involving claims of misappropriation of trade secrets, breach of contract, and conversion. Plaintiff argued that sanctions were warranted because of Defendant’s misrepresentations concerning the late source code production and Defendant’s failure to produce “smoking gun” email, which was produced by another party to the litigation. The other party was the recipient of the “smoking gun” email. Defendant argued that its computer system did not retain copies of any sent emails.

The court allowed an adverse inference instruction and monetary sanctions based on the belated production of source code and the misrepresentations about the existence of other versions of the source code. The court did not grant sanctions based on the missing “smoking gun” email because the court reasoned that the automatic deletion of the email was not done in bad faith. Instead it was their usual policy, albeit an uncommon one. The court did, however, question the wisdom of a business not saving any of its sent email, but that was beside the point for sanction analysis. Here is the Court’ ruling:

For any business this is a significant irregularity; almost unimaginable for a technology company; and even more unlikely for a person of Bowers’ importance in such a company. Nonetheless, it does not appear that the September 5, 2005, email was withheld by WideBand Defendants-they did not have any copies of emails sent by Bowers.

WideBand Solutions, Inc., did not maintain an email storage system that would retain a copy of the September 5, 2005 email. No evidence suggests that this was done in bad faith, but is rather the effect of design of the email system WideBand employed. However questionable the design may be, the effect is that the routine operation of the WideBand computer system did not capture the email. No sanction is needed on this point, as ClearOne is free to establish at trial that no one has complete access to or knows the entire contents of Bowers’ sent email. Each party will be free at trial to argue the implications of that fact.

The court did not mention Rule 37(e), but this case is a good example of where the rule would apply. Here is the wording of the Rule:

Electronically stored information. Absent exceptional circumstances, a court may not impose sanctions under these rules on a party for failing to provide electronically stored information lost as a result of the routine, good faith operation of an electronic information system.

That is exactly the situation the court describes in this case and so the refusal to impose sanctions here complies with the new rule.

Coburn v. PN II, Inc., 2008 WL 879746 (D. Nev. Mar. 28, 2008). In this employment discrimination action, Defendants sought to obtain a forensic examination of Plaintiff’s home computer. Defendants stated that the search would focus on information regarding Plaintiff’s employment with Defendants, the employment termination, Plaintiff’s claims, and possible damages. They offered to pay for the whole thing. Surprisingly, Plaintiff did not object to the forensic examination, but she wanted it to be a more “limited, focused” inspection. Plaintiff also opposed the request because she claimed Defendants did not set forth a protocol or methodology that would protect her against violations of privilege, privacy, and confidentiality interests.

The court did not really address Plaintiff’s scope concerns, but instead generally stated that the burden on Plaintiff was minimal, and noted that Defendants had agreed to pay for the imaging. The court cited Playboy Enterprises, Inc. v. Welles, 60 F.Supp.2d 1050, 1054-55 (S.D. Cal.1999), with approval as establishing a good protocol to protect confidentiality. The court then followed Playboy and set forth its own detailed procedures to protect Plaintiff’s privacy rights.

The case is interesting for the detailed protocols ordered. It is also unusual in that the inspection per se was unopposed. I expect many will gloss over this key fact, and try to use Coburn to argue for a right to inspect another’s computers, so long as privacy is protected, and the inspection is paid for. That is contrary to the law, where good cause or other unusual circumstances must be shown, such as discovery abuses, or evidence of missing ESI.

Daimler Truck N. Am. LLC v. Younessi, 2008 WL 2519845 (W.D. Wash. June 20, 2008). This case alleges breach of duty of loyalty and confidentiality based upon an employee’s leaving to go work for a competitor, and allegedly disclosing confidential information to the competitor. Defendant Ramin Younessi was a “high level executive” of plaintiff, Daimler Truck, before going to work for a competitor truck manufacturer. Daimler served a subpoena upon a truck dealer, Cascadia International LLC (”Cascadia”), who was not a party to this litigation. The subpoena requested production of Cascadia’s computers so that Daimler’s experts could search them for certain communications involving the alleged confidential information. Cascadia of course moved to quash the subpoena and for a protective order because the request was unduly burdensome and required disclosure of its trade secrets. No doubt you have already correctly deduced that Cascadia is not a Mercedes truck dealer.

The court did not entirely quash the subpoena because the requested communications were highly relevant. However, the court did not allow Daimler to inspect the computer hard drives because of the risk of disclosing trade secrets. The court ordered Cascadia to search its own computer systems and produce any relevant communications. In response to the objection that the search was unduly burdensome because of the many locations and computers involved, the court directed Cascadia to identify all sources that it did not search, and all potentially relevant documents that it did not produce, in order to evaluate the costs of providing the documents and likelihood of finding relevant documents. The court also limited the production to documents found on certain hard drives and other electronic devices.

Although the Court here reached the right result, the opinion is remarkable for it’s misunderstanding of the rules and the nature of ESI. Read this language of the decision and see if you can tell what is wrong.

The Federal Rules provide for discovery of electronically stored information either in its original state, i.e. actual production and copying of hard drives, or in a reasonably usable form, i.e. print outs. Fed.R.Civ.P. 34(a)(1)(A). . . . That is, the Rule allows for a subpoena of an entire hard drive for the limited purpose of finding a few documents which may be stored therein. See Fed.R.Civ.P. 34(a)(1)(A) (requesting party may obtain information stored in any medium); Fed.R.Civ.P. 34(b)(1)(C) (requesting party “may specify the form or forms in which electronically stored information is to be produced”). This would be analogous to allowing the search of a party’s entire collection of file drawers for the purpose of finding a single class of documents.

That is not what the rules mean at all. The rules allow for the request and production of relevant electronically stored information. ESI is not equivalent to hard drives. The original state of ESI is not hard drives. It is digital information. ESI is sometimes stored on hard drives. It lives there sometimes, it is it’s “house” so to speak. But it is not the same thing as the house itslef, anymore that I am. If you want to see the original of me (dubious desire) you do not subpoena my whole house and then go in there and look around for me. You subpoena me, and out I come. It is the same way for ESI. The original state of the ESI is not the same thing as the “hard drives” or other media in which it lives.

The ESI they were searching for in this case was computer files, most likely email. These files that can be stored on any device. When the original ESI is copied from one media to another, if it is done right, there is no alteration at all. It is still the same original ESI, it is just located on another medium. In that regard, ESI is just like a person. It is still the same person when he or she moves out of the house and into a car. It is also the same original ESI regardless of whether it is found on a hard drive, DVD, thumb drive, iPhone, or whatever.

It is obvious from the language that this fundamental nature of ESI has been misunderstood, and this misunderstanding then leads to needless machinations and worries about the rules. They are better written than that, and the revisions of December 1, 2006, do not, analogously speaking, turn common sense on its head and now allow a party to search another party, or worse, a third party’s “entire collection of file drawers for the purpose of finding a single class of documents.” People search their own drawers, thank you, just like they always have.

Ed Schmidt Pontiac-GMC Truck Inc. v. DaimlerChrysler Motors Co., LLC, 538 F. Supp. 2d 1032 (N.D. Ohio 2008). This is a case by an automobile dealer against its manufacturer, Daimler-Chrysler, alleging breach of a settlement agreement. The opinion addresses the dealer’s motion for leave to amend the complaint to add a new complaint under state law (Ohio) for spoliation. Plaintiff alleges that Daimler-Chrysler lost evidence by not instigating a proper litigation hold after the complaint was filed, and by intentionally destroying evidence by replacing employee computer hard drives only days before Plaintiff made forensic images of them. The court granted Plaintiff’s motion holding that Plaintiff presented sufficient evidence to allow the claim to be pled.

One of the arguments Daimler-Chrysler employed to try and defeat this motion was that arguments on this spoliation claim to the jury would be more prejudicial than an adverse inference instruction. The Court disagreed, and noted (correctly, I think) that such an instruction regarding destruction of evidence would also be extremely detrimental to the defense. Note that this maneuver, to add a new claim for spoliation, instead of just seeking sanctions for spoliation in the underlying claim, is not available in many states, such as Florida, that do not allow a separate tort for spoliation in these circumstances.

Ferron v. Search Cactus, LLC, 2008 WL 1902499 (S.D. Ohio Apr. 28, 2008). Plaintiff, John W. Ferron, is an attorney who brought an action for himself under the Ohio Consumer Sales Practices Act for unlawful, deceptive spam. Ferron claimed that the emails he received from Search Cactus were misleading and deceptive, in that they offered “free products” without adequately disclosing the conditions attached to claiming those purported “prizes.” The unsolicited emails Ferron received were the main evidentiary support for his claim.

Search Cactus requested an inspection of Ferron’s computer systems to determine whether the emails and website visits were a legitimate consumer transaction under the statute. They in effect argued a kind of entrapment, that Ferron induced poor Search Cactus into spamming him, and then sued them for it as a consumer violation. Apparently Ferron admitted to filing many such suits for himself, and for his clients; to which I say, what’s wrong with that? Many people dislike spam and everybody needs a hobby.

Search Cactus wanted to inspect Ferron’s computers to seek information it said would allow it to determine “whether Plaintiff’s opening of the emails and any attempts to obtain free merchandise were part of a business designed to profit from email litigation.” Search Cactus was looking for the Internet browsing history shown by the computers, as if this would somehow prove their point. The opinion does not explain this, or many other technical aspects.

This case is again unusual in that Ferron, an attorney, does not oppose the inspection, or at least, this battle has been previously fought and lost for reasons not explained in this opinion. Instead, Ferron seeks only to limit the forensic inspection to one small segment of his computers. He argues that the Internet browsing history is stored in only one directory (typically true), but for some reason, again not explained, does not simply produce this information. Search Cactus disagrees that the history is only in one location, and instead argues that complete images of the entire drives are necessary to find the browsing history. Apparently, Search Cactus suspected Ferron of hiding the history in other areas of the drives and of deleting it, or at least trying to.

The court allowed a full forensic examination, but provides only vague justifications, holding that the inspection of the computers was necessary because: (1) there was evidence that Ferron had failed to preserve relevant evidence (the opinion does not explain what was supposedly destroyed); (2) Ferron had not produced the relevant information (again, that is a mystery - finding and producing the history should be easy, and it is hard to understand why this was not done by Ferron long ago); and (3) Ferron’s computer system was the only place where the evidence requested was contained (well yes, but again, it could easily be copied and live on a production CD as well). The court recognized Ferron’s concerns regarding confidential and privileged information. For that reason, it issued a Playboy type protocol to protect Ferron’s rights during the computer inspections. No word on what protection he was offered against tricky spam.

Flagg v. City of Detroit, 2008 WL 787061 (E.D. Mich. Mar. 20, 2008). Detroit seems to be having more than its share of city employee text message problems. The sexual text messages and cover-up of the infamous Mayor of Detroit are well known. But the Flagg case, again involving text messages by city employees in Detroit, is not. Flagg concerns a more serious subject than a politician’s sex life, and the court established a detailed protocol for reviewing text messages between a large number of city officials.

The Plaintiff in Flagg is the minor son of a murder victim. The murder remains unsolved. Plaintiff alleges that the employees of the city did an incomplete investigation of the murder, and even intentionally ignored and concealed significant evidence. Plaintiff alleged that this negligence and willful misconduct resulted in Plaintiff not being able to bring a wrongful death lawsuit against the murderer.

In a previous order, the court denied Defendant’s motion to quash a subpoena given to SkyTel to produce text messages between city officials. The court found that the text messages might contain discoverable information, thus Plaintiff was allowed to pursue inspection of these messages. The order established a protocol where the PIN numbers of the text messaging devices of certain city officials were given to SkyTel to retrieve the messages. Two magistrate judges were assigned to oversee and control the receipt of the text messages from SkyTel, review the text messages, and determine if certain text messages are discoverable under Federal Rule of Civil Procedure 26(b)(1).

Grange Mutal Casualty Co. v. Mack, 2008 WL 744723 (6th Cir. March 17, 2008). This is a Sixth Circuit opinion affirming a default judgment entered as a sanction for a host of discovery abuses, both paper and electronic. Several insurance companies sued Greg Mack in District Court in Kentucky. Mack was an operator of medical clinics that specialized in the treatment of automobile accident victims. The insurer Plaintiffs sued Mack for fraud and Racketeer Influenced and Corrupt Organizations Act (RICO) violations. In the words of the Sixth Circuit, “Mack bilked the companies by setting up medical clinics to treat auto accident victims and then using those clinics to diagnose phony injuries and overcharge the companies for the needless medical services performed.”

The district court ordered a default judgment for liability and damages as a sanction for Mack’s many egregious discovery abuses, including a few e-discovery abuses, such as secretion and destruction of computers. Mack appealed to the Sixth Circuit, and not too surprisingly, in view of the incredible history of defiance and refusal to provide discovery, the default sanction was easily affirmed. The Court opinion recounts some of the most damning evidence against Mack, including testimony from one of his former attorneys. Here is an example from the opinion:

Although our court hesitates, but is not entirely unwilling, to approve a default judgment when any misconduct is solely the fault of the attorney, Harmon, 110 F.3d at 367-68, we need not hesitate here because Mack perpetrated the discovery abuse himself. Mack personally instructed Dr. Santelices not to produce numerous discoverable documents and personally fired Dr. Santelices when the doctor turned over a complete copy of his date book instead of the edited version that Mack initially turned over. Mack personally changed the office computers despite pending litigation, refused to turn over other computers, and issued blanket, frivolous objections to the Grange plaintiffs’ discovery requests. Prior courts have treated similar frivolous objections as evidence of willful conduct. Bank One, 916 F.2d at 1074-75.Furthermore, Mack’s former counsel, Robert Riley, admitted that important documents had been removed from his files. Documents do not jump out of file boxes on their own.

The final words of the Sixth Circuit in Grange are well put, and I predict will be quoted frequently by litigants seeking sanctions for discovery abuse:

Our civil legal system hinges on voluntary discovery. Discovery abusers must be sanctioned, because “[w]ithout adequate sanctions, the procedures for discovery would be ineffectual.” 8A Wright, Miller & Marcus, Federal Practice and Procedure § 2281 (2d ed. 1994). Judge Hood acted well within his discretion in ordering the default judgment. We affirm, both to punish Mack for his egregious conduct and to deter other litigants who might be tempted to make a mockery of the discovery process. See Nat’l Hockey League v. Metro. Hockey Club, 427 U.S. 639, 643, 96 S.Ct. 2778, 49 L.Ed.2d 747 (1976) (per curiam).

 

What Game Does an e-Discovery Team Play?

Hide the ball is certainly not the game for an e-Discovery Team to play. Some people think that is what discovery is all about, and in the world of paper discovery, years ago, there was some truth to that. But not today, and certainly not in electronic discovery. It may be tempting to some, but if you play hide the ball in e-discovery, and get caught, you may not only lose the case, but you may lose your job, and maybe even your license. It is never worth it, just ask Qualcomm’s lawyers. Instead, an e-Discovery Team plays a series of games that culminates in throwing the ball to the other side, not hiding it.

Before you can get to the final throwing step of production of electronically stored information (”ESI”), there are a series of preliminary games to be played. Here is how I summarize the e-Discovery team playbook:

Find the Ball

Save the Ball

Pick up the Ball

Shrink the Ball

Clean the Ball

Aim the Ball

Throw the Ball

The first game of find the ball is called the Identification step in the standard industry language of the Electronic Discovery Reference Model(”EDRM”). By looking at the standard EDRM model below you can quickly see how each game represents a basic step in the EDRM.

Find the Ball

Finding the ball is far easier said then done. For most companies, the problem derives from storing terabytes of data. Imagine a string of warehouses storing a billion basketballs, and you have to search through and find the one ball among them autographed by Michael Jordan. Unless the Team is well established, you probably do not have an accurate, detailed, up-to-date map of all of the warehouses. You probably have only a vague idea where this one basketball might be located. It might be somewhere in a centralized bin, or in any one of dozens of other locations, including closets in employee homes, or off-site Internet storage lockers. It might even exist only in a shrunk down version, hiding in the pocket of one of a thousand employees; perhaps in their thumb-drive, or iPhone. Moreover, ever day a thousand basketballs are destroyed (hopefully not the one with Jordan’s autograph), and twelve hundred new ones are added. Yes, it is a very challenging game indeed

To make matters worse, you are never sure exactly what balls you are looking for, especially when the game first begins. You may have to guess, from a vague complaint, what balls are relevant. As I have written before, this is one of Anne Kershaw’s pet peeves, and rightfully so. Under federal notice pleading rules, very few details are required in a complaint to state a cause of action. So defense counsel is often left speculating what ESI will be discoverable and relevant in a new case. Still, you have to start making educated guesses to try to find the right batch of balls. From the large selection first identified, you will eventually throw a few to the other side.

The way most teams do this is to analyze the dispute to try to determine what the issues will be in the case. This gives you a general idea of the types of balls that may come into play. Then you start to determine a general time line; hopefully the potentially relevant balls will be constrained by time. You may be able to know, with some certainty, that balls made before or after a certain time are not relevant and need not be searched. An e-Discovery Team will also try to limit the search to balls made or stored by certain key players. These are the people in your company that are likely to be involved as witnesses in the lawsuit. The Team’s search should be focused on the storage bins of these key players.

Save the Ball

After playing find the ball, the next game is save the ball. Here the Team devises ways to preserve most of the balls identified as potential evidence in the last game. Again, this can be a very challenging game, especially when your company has many different auto-destruct routines in place (and most companies do).

If you think it is easy to stop all of these programs, just ask Intel. They thought they had stopped deletion of excess email for all the key players in the anti-trust case against AMD, but in fact the janitor programs remained in place for the most important players, including the top officers of the company. Their email was deleted for years after the case was filed. They tried to play a very complicated game of save the ball, but failed. For a better idea of just how difficult this game can be, check out Intel’s report to the supervising district court judge on their failed attempts to preserve evidence. This mistake has supposedly already cost Intel millions of dollars to correct by forcing them to go to their backup tapes to find the deleted emails, and the meter is still running. AMD is, of course, claiming that the error was intentional. They would like the court to enter sanctions for spoliation and turn this mistake into an outright win of the whole case.

So make no mistake about it, save the ball is one of the most important games an e-Discovery Team plays. As I have discussed before, that is why most e-Discovery Teams focus on this game as soon as the Team is formed, and look for ways to improve their company’s litigation hold procedures.

Pick Up the Ball

Again, this game sounds easy enough, you just collect the relevant ESI from the data you have identified and preserved. Seems easy, but it is not. There are tricks and traps here aplenty. If you are not careful, you could collect too much or too little. Generally you do not want to simply pick up all of the balls you have saved. That will make the next games too expensive. You want to screen out the ones that are unlikely to be needed, and probably are not relevant at all, but were preserved just in case. You want to preserve more broadly than collect because you never want to play save the ball twice in the same case. Not only is that kind of do-over expensive, but it may be futile because, in the meantime, routine processes may have deleted many balls not saved in the first pass.

You also do not want to pick up too few balls, and leave behind many that are directly relevant and should later be thrown to the other side. That kind of careless collection can also be expensive. It can force you into an expensive do-over, and open you to charges of hiding the ball. See Eg. Court Disapproves Defendant’s “Hide the Ball” Discovery Gamesmanship.

Careless collection often occurs if the Team simply delegates this function to the key witnesses, and does not properly supervise or follow-up on their ball-picking efforts. The same comment holds true to the two prior games of ball identification and preservation. The Team cannot over-delegate its responsibility to key players and then just hope for the best. These are their games, and the Team must take responsibility to see they are played correctly. That is the whole purpose of an e-Discovery Team.

For that reason, in most cases it will not suffice to simply send out a preservation letter to the key players which describes the dispute, and then leaves it to them to find the relevant balls for themselves, save them, and pick them up. Without help and supervision from the Team, the key players may not know which of their computer files are relevant, they may not know how to properly preserve this ESI, nor how to collect it. They are sure to make mistakes. Thus, when the key players in a company are called upon to take part in the games, which in itself makes a lot of sense, since they should know their own information better than anyone else, they should be given expert help and advice from the e-Discovery Team. In other words, it is perfectly all right for the Team to delegate some of this work to the key players in the litigation, but the Team must still supervise and follow-up. Ultimately the Team should be responsible, since they are trained and more experienced in collection than the key players. The Team should have personal meetings with the key players and closely monitor their activities. In many cases, the Team should also implement certain safeguarding mechanisms to supplement the key players’ efforts, such as automated copying and keyword searches.

Another common mistake made in pick up the ball is to carelessly change the ball in the very process of picking it up. You could, for instance, change the metadata of a file, such as information as to when it was last viewed, saved, or revised. This is an especially high risk when the Team attempts to rely upon key players to pick up the ball for them. Although this probably will not matter in most cases, in some cases, such as stock backdating cases, this might be very important. As a general rule, the Team tries not to change the ball too much by the act of picking it up. The Team may later strip a file of all or part of its metadata on purpose, if that facilitates later cleaning or throwing, especially if the metadata is not important in the case, or not wanted, but they never want to do it accidentally.

A final common mistake, one of my pet peeves, is to neglect to hash the ball when you collect it, and properly preserve and tie the hash into each ball thereafter. I have described the process of using hash mathematics to authenticate ESI at length in my law review article, HASH: The New Bates Stamp, 12 Journal of Technology Law & Policy 1 (June 2007). I also provide an overview of the subject in this blog. The Team may already have hashed files as part of the preservation game; but if not, it is essential that they now be hashed at the collection stage. Hashing provides a unique identifying alphanumeric value for each computer file collected. This hash value can be later checked to prove that the file has not been altered since it was collected. This is a key step in ESI authentication to allow for admission into evidence at a hearing or trial. In most cases, hashing should be a normal part of ball pickup.

Shrink the Ball

Shrink the ballis the game where the Team can save the company a lot of money. Thus, from a financial perspective, it is the most important game of all. In this culling step, you process the ESI to eliminate as much duplicate and irrelevant information as possible. Here good software and automated process are critical; so too is careful strategic thinking,

The goal is to significantly reduce the amount of ESI that must be reviewed and cleaned in the next steps. Thus, for instance, at the end of the last game you may have identified and preserved 1,000 gigabytes (1 terabyte) of ESI, and collected 500 gigabytes. To give you some idea of the amount of information we are speaking about, in some circumstances the 500 gigabytes may be equivalent to 500 truckloads of paper. It would cost a small fortune for teams of lawyers to read that much paper. We are talking about years of billable lawyer time to read that much data. It would also be a colossal waste of time because they would end up reading the same document dozens, if not hundreds of times. So it is critical to aggressively eliminate the redundant and immaterial ESI in this processing stage. In many cases the 500 gigs can be cut down to 100 or 50 gigs, resulting in tremendous savings in the expensive review games to come.

Clean the Ball

Here is where the big bucks come in, the cost to review the data for privileged, confidential, and irrelevant material. Still, most internal corporate e-Discovery Teams will not clean their own ball, they will hand it off to their caddy to do it for them, typically their outside legal counsel. A few of the more mature and well organized Teams have started to review their own data, and clean them the ESI themselves. They have teams of contract attorneys they employ to do this work at reduced rates, some even send the data to lawyers in India for review. But for most Teams, this is advanced play that they do not have the time or skill to attempt.

This is a very important and risky step in the EDRM process and companies want to be sure it is done right. You review the truckloads of email and documents that have not already been culled out in the prior games so that you can remove the files that do not have to be produced. The last thing you want to do is produce privileged materials to your adversary. You need to clean your production of these secret files and produce a log of them instead. Even with a clawback agreement, an accidental disclosure can still result in waiver of your privilege to third parties. You also want to be sure the ESI review catches all confidential materials, and that they are produced with appropriate markings and confidentiality agreements. Trade secrets can be lost forever if they become a public record by filing with a court.

Aim the Ball

Now we come to the lawyerly game of aim the ballwhere the ESI is analyzed to see how it fits into the case at hand. Here lawyers and paralegals tag each file to an issue, typically using review software. They also make final decisions as to whether and how information is responsive to discovery requests, or otherwise must be produced (or not). The files are categorized and rated for importance. Is this email a smoking gun that could kill your case, or is it merely of marginal relevance to a secondary issue? You had better find this out, and fast, as to each computer file you are about to disclose to the other side. If your analysis of the information to be produced shows you have a strong case, you will approach the case far differently than if your analysis shows you will surely lose when all of the cards are put on the table.

Obviously this analysis stage requires the sure hand and steady aim of trusted outside counsel assigned to defend or prosecute the case. Still, the legal members of the Team should assist and be involved in the analysis and evaluation of the merits of the case. This game concludes with final decisions by legal counsel on what should be produced and what should be withheld. These decisions must be rational and made in good faith.

If analysis shows you have a losing hand, you had better fold early before the other side realizes your position. You cannot do like Qualcomm and decide to withhold evidence just because you don’t like it. You can see where hiding the ball got them - they lost the patent they sued to enforce, they paid over eight million dollars in fees to the other side, their general counsel resigned in disgrace, and their outside counsel are now fighting to retain their licenses. When you are a plaintiff and find yourself in this position, you do not file the suit to begin with or, if you discover it in midstream, you should dismiss and cut your losses. The same applies when you are in a defense position. It is not an option to try and hide the evidence that will hurt your defense. You must instead try and make the most of it and settle as best as you can. That is how the American system of justice works and all Teams have to play by these same fundamental rules. Voluntary disclosure may not be the rule in the rest of the world, especially the civil law countries in Europe, but that is how the game is played here. If you are defending or prosecuting a case in the U.S., you are going to have to reveal your data to your adversary, even if that kills your case.

Throw the Ball

The last game is the culmination of all the rest. The analysis game resulted in final decisions on what files to be produced. Now you actually make the production. Throwing the ball is not really all that hard, so long as you enlist the aid of WORMs. No, not the creepy crawly kind, but the “Write Once, Read Many” times kind, such as optical discs, CDs or DVDs. The ESI on these media cannot be altered after written onto the discs, thus providing you, and the receiving party, with a certain amount of protection that the files will not be accidentally altered. Worms help the parties maintain a permanent record of the ESI produced.

Another tricky aspect of production is deciding the form of production. Do you produce in native format with full internal metadata retained, or do you produce in a TIFF or JPEG format with a load file ready for import into review software? This should have already been worked out with opposing counsel as part of the Rule 26(f) conference, or the original production request; but if not, you have to make these decisions now.

Take the time to clearly mark and label the production media. One thing I hate is a CD production with no writing on it, or just indecipherable handwriting. Write out a full description of the CD and the date of production and name of the case. Think of chain of custody issues and do not forget to make multiple copies. Another thing I have noticed lately is the use of paper labels on CDs. That’s ok, but beware of labels that peel off. As a safeguard, it is better to use ink jet printers that print directly on the CD, instead of glue on labels. If you must use adhesive labels, put some kind of writing directly on the CD itself, just in case it peels off somewhere down the line.

Finally, if you use TIFF or other image type files where you affix Bates stamp type markings to identify individual ESI files, please consider adding a truncated hash value to the file ID. As discussed in HASH: The New Bates Stamp, this will facilitate both identification and authentication, and allow for easier comparison with the native originals.

Concluding Thoughts

These games are difficult. Much like golf, it is not a game of perfect. Mistakes are inevitable. Even Tiger Woods messes up from time to time, and does not win them all, so why should you be any different? Document your efforts, play it safe, and use redundant systems whenever economically feasible. Thus, when a mistake is later discovered, you may be able to cover it with a backup system. Or, if that is not possible, you can at least show to the supervising judge that you made good faith, reasonable efforts. The judge should understand and cut you a break, maybe even give you a mulligan. If the judge does not realize that mistakes are inevitable, he or she simply does not understand the game. Then it is up to you to explain it to them, or hire an expert who can.

Venue Analysis Transformed by e-Discovery and the Digitization of Society

The location and availability of documents have always been important considerations in determining whether venue should be transferred for “the convenience of parties and witnesses, in the interest of justice.” 28 U .S.C. §1404(a). Not any more. Two new cases have shown that the digitization of records and electronic discovery are quickly rendering these criteria obsolete. Victory Intern. (USA) Inc. v. Perry Ellis Intern., Inc., 2008 WL 65177 (D.N.J., Jan. 2, 2008); ICU Medical, Inc. v. RyMed Technologies, Inc., 2008 WL 205307 (D. Del., Jan. 23, 2008).

When parties argue about venue, and what court is better situated to hear a case, they argue about convenience. These arguments not only include the location of witnesses, but, traditionally, also the location of records. If a court is located closer to where the original paper records are stored, that is supposed to be a factor favoring selection of that court.

This law developed in the last century when almost all documents were paper. At that time it made sense. If a case involved a warehouse full of paper records, then the proximity of a court to that warehouse was an important consideration. It made document productions, depositions, hearings and trial easier and less expensive. The proximity to original records was one of several factors of convenience and justice that a court would consider in determining whether to transfer venue.

In this new century, well over ninety percent of business and other records are now in digital form. The electronically stored information can be fairly easily, and sometimes almost instantly, transferred from one computer to another, regardless of where the computers are located. With hash verification in place, the electronic document in the second computer is just as much an original as the electronic document in the first. It is exactly the same computer file. Paper printouts of these multiple original electronic records can then be made anywhere on demand. For instance, they can be made if and when needed for depositions, hearings and trial. The proximity of the courthouse to the location of the computers used to create and store the information is irrelevant.

Victory Intern. v. Perry Ellis Intern.

The Victory case involves a fight over perfume and the right of Victory International to distribute Perry Ellis fragrances. Victory Intern. (USA) Inc. v. Perry Ellis Intern., Inc., 2008 WL 65177 (D.N.J., Jan. 2, 2008). It was brought by a single plaintiff in New Jersey against a host of companies and individuals. As is common in anti-trust type cases like this, the plaintiff, Victory, alleged a long list of complaints:

Victory seeks relief against the defendants for violation of Section 1 of the Sherman Act (15 U.S.C. § 1), violation the Donnelly Act (N.Y. Gen. Bus. Law §§ 340-347), interference with contract, interference with prospective business advantage, breach of contract, fraud, deceit, unjust enrichment, violations of the Florida RICO statute, deceptive trade practices, restraints of trade, and unfair competition under the common law of the State of New Jersey and the other states of the Union.

The plaintiff, and a couple of the defendants, were located in New Jersey where the suit was filed, but thirteen of the defendants were located in South Florida. The defendants moved the District Court in New Jersey to transfer the case to the District Court in Miami. The parties all agreed that both courts had jurisdiction, so it was strictly a venue issue.

Senior New Jersey District Court Judge Walls begins his analysis with the federal statute governing venue transfer: 28 U .S.C. § 1404(a). It states that if two District Courts have jurisdiction, then for “the convenience of parties and witnesses, in the interest of justice, a district court may transfer any civil action to any other district or division where it might have been brought.” Judge Walls goes on to explain:

A determination of whether to transfer must incorporate “all relevant factors to determine whether on balance the litigation would more conveniently proceed and the interests of justice be better served by transfer to a different forum.” Jumara v. State Farm Ins. Co., 55 F.3d 873, 879 (3d Cir.1995) “Transfer analysis under Section 1404 is flexible and must be made on