New Case where Police Use Hash to Catch a Perp and My Favored Truncated Hash Labeling System to ID the Evidence

Part of my discipline as an e-discovery specialist is to try to read (or at least skim) every published opinion on the subject. Lots of attorneys specializing in this area do that. But there is one other type of case I also read, every opinion that uses the word “hash.” No, I do not need help from Narcotics or Overeaters Anonymous. The kind of hash I am addicted to is purely algorithmic. This hash comes in many flavors, but the best known, and the ones usually employed in e-discovery, are called MD5 hash, SHA-1 hash, or the latest and greatest, SHA-2 hash. 

As I explain in my blog Hash page, hash is the mathematical foundation of e-discovery and the most powerful tool of any forensic investigator. It reveals the unique mathematical fingerprint of every computer file that allows for perfect identification and authentication of electronic evidence.  I became fascinated with the powers of hash a few years ago, and ended up writing a lengthy law review article on the subject. HASH: The New Bates Stamp, 12 Journal of Technology Law & Policy 1 (June 2007). A few months ago I wrote a blog on the article called The Days of the Bates Stamp Are Numbered, talking about some of the more recent developments in this area of the law, especially the shift from Tiffing and linear flat file Bates stamping to native file hash marking.

In the process of researching the original law review article, I am pretty sure I read every legal opinion and legal article ever written that  mentions hash. I also read a few scientific and cryptological articles as well, most of which I did not really understand. Having put that much time and effort into the subject, I try to keep up by reading every new legal opinion or article mentioning hash. That is why I have a standing search for all cases using the term, and automatically receive a copy of them by email as soon as they are published. I can be in the middle of dinner and my blackberry will buzz alerting me of a new hash case. Lest you think that’s a tad weird, I am willing to bet that there are a few other hash enthusiasts out there, Craig Ball comes to mind, who do the same thing. (See Craig Ball’s excellent article “In Praise of Hash” at pg. 52.)

Hash and Child Pornography

Most of the new hash cases I see have nothing to do with e-discovery per se. Instead, they are usually criminal law cases, typically cases involving one of the most disgusting of crimes, child pornography. Police have been using hash to catch perps in this area for years. Hash is an effective tool for this because it allows police to know if certain child pornography is located on a computer, usually videos or still photos, by looking to see if the hash values for these files are present. That is a bit of an over-simplification, but suffice it to say that there are lists of hash values that are known to be associated with computer files which are unquestionably child pornography. New York Attorney General Andrew Cuomo explained the process in a press release in June 2008 announcing a deal with major Internet providers to block major sources of child pornography: 

As part of the undercover investigation, the Attorney General’s office developed a new system for identifying online content that contains child pornography.  Every online picture has a unique “Hash Value” that, once identified and collected, can be used to digitally match the same image anywhere else it is distributed.  By building a library of the Hash Values for images identified as being child pornography, the Attorney General’s investigators were able to filter through tens of thousands of online files at a time, speedily identifying which Internet Service Providers were providing access to child pornography images.

U.S. v. Warren

I recently received a new hash case alert from a district court in Missouri. U.S. v. Warren, 2008 WL 3010156 (E.D.Mo. July 24, 2008). A quick review showed it was yet another child porn case, so I did not think much about it. I just added it to my reading list for more careful study later, just in case there might be something special about it. When I got around to reading Warren yesterday, I was very pleasantly surprised, as this was indeed a special case.

Warren is a case considering and rejecting a motion to suppress evidence, namely computer video files of underage teens having sex. The motion to suppress was based on a series of hyper-technical challenges to the affidavit which the St. Louis police submitted to the judge to receive a search warrant of defendant’s computer. The affidavit explained how the police had searched the Internet for files “whose digital SHA-1 value was identical to that of a file known to contain child pornography.” They found a computer with an Internet Protocol address of 70 … 167 offering to share one such known file, and then subpoenaed AT&T to get the physical address of the subscriber with that IP address. The computer was located in Affton, Missouri.

The police detective’s affidavit explained how the hash values and offer to upload established “that a computer in Missouri was ‘offering to participate in the distribution of known child pornography.’” Based on this affidavit, the judge found probable cause to issue the search warrant of the computers located in Warren’s home. The police then went to his home, found no one there, forced entry, and seized his computer. Warren himself later came along, and, foolishly enough, voluntarily came to the police station, waived his right to counsel several times, and spoke at length to the police. The opinion includes extensive excerpts of the taped interview, which Warren later argued was made in violation of his right to legal counsel.

The defendant’s technical search warrant objections forced the court to delve into many of the characteristics and evidentiary properties of hash. For that reason alone, the case is useful to any practitioner trying to better understand the subject. But what is really special about the case, at least for me, is the system of hash file identification used by the court to identify the offending video tape at issue in this case. That video computer file was the key piece of evidence, the “smoking gun.”

Six-Place Hash Truncation Naming Protocol

The opinion by Magistrate Judge David D. Noce in Warren is unusual and special because it is the first case to use the truncated hash value labeling system I proposed in HASH: The New Bates Stamp. My article was not mentioned, and apparently Judge Noce was not aware of it. He used the six-place hash truncation system I proposed in my article because it was, in his words, “convenient” to do so, and because the detectives had used that system in their affidavits and testimony. I doubt the police detectives had read my law review article either, which makes their use of the abbreviation system all the more important. It shows that it is a natural and reasonable thing to do, although this is the first time it has been utilized or mentioned in a legal opinion.

So what is the six-place hash truncation system which I proposed that these Missouri officials are now in fact using? Before I can answer that, I have to go into a little more depth about hash and Bates stamps. HASH: The New Bates Stamp not only explains hash and its importance to e-discovery, it also argues for the legal profession and e-discovery industry to adopt a new type of electronic document naming protocol that uses hash values, instead of sequential numbering, to identify electronic evidence. I argue that the time has come for the legal profession to abandon Nineteenth Century Bates stamp paper mentality, and adopt Twenty-First Century ESI hash mentality. I proposed that sequential Bates stamps be replaced by non-linear, intrinsic hash values.

The hash values would not only identify ESI, they would authenticate it too, something the lowly Bates stamp could never do.  But the problem with using hash values to identify ESI, instead of Bates stamps, is that hash values are too long and awkward for the human mind. Here is what a typical forty place hexadecimal SHA-1 hash value looks like: 2B37BC6257556E954F90755DDE5DB8CDA8D76619.

Police detectives, lawyers and judges cannot go around describing computer files used as evidence with such long alphanumerics. It is too cumbersome a name to replace the Bates stamp. So my common sense proposal, which Judge Noce in Warren calls “convenient,” is to only use the first and last three places of the hash value, instead of all forty. So the hash value above becomes the much more manageable 2B3 … 619. That truncated hash value becomes a pretty good document name, and, in my opinion and that of many others, should replace the arbitrary Bates stamp.

Turns out that the detectives in Missouri were already following this six-place truncation protocol at the time my article was published in June 2007. Perhaps they and other law enforcement agencies have been using this system for years. I do not know for sure, although I doubt it has been a widespread practice. I have talked to many e-discovery forensic experts about the hash naming proposal over the past two years. Many of these experts did police work before going into e-discovery, and none ever mentioned having done this before. Also, it certainly does not appear in the legal literature on the subject, that is, until U.S. v. Warren.

Hexadecimal Values v. Base32 Number System

At first, I was disappointed to see that Judge Noce’s introduction of the truncated hash value naming protocol was flawed with two obvious technical errors. See if you can catch them:

The search turned up a list of files, including one with a 32-character alpha-numeric SHA1 designation of “H4V … UTI.” Fn4

FN4 - For convenience, in this opinion the SHA1 value set out in full in the search warrant affidavit will be referred to as “H4V … UTI.” The affidavit defined the term “SHA1” (also known as “SHA-1”) as being a mathematical algorithm that uses the Secure Hash Algorithm (SHA), developed by the National Institute of Standards and Technology (NIST), along with the National Security Agency (NSA) . . . Basically the SHA1 is an algorithm for computing a condensed representation of a message or data file like a fingerprint.

Warren at *1.

First of all, the SHA-1 hash generates a 40-character hexadecimal string, not 32-character. The other kind of hash, MD5 hash, is the one that uses a 32 character string, not SHA-1. For this reason, my first reaction was that the Judge, or police, mixed up the two different types of hash, and meant to say 40 characters, not 32.

But then there seemed to be yet another, even bigger mistake. The letters H V U T and I should not have been in the hash value name. The values generated in e-discovery work to represent SHA-1 and MD5 hash are always hexadecimal. That is a numerical system with a base of 16. This is typically represented by the numbers 0–9 for the first ten values, and A, B, C, D, E, and F to represent the last six, for a total of sixteen. In other words, a hexadecimal value does not employ any letters after F. Yet, the so called SHA-1 alphanumeric stated in the Warren opinion uses the letters H, U, T and I: “H4V … UTI.”

I thought the police or Judge Noce must have messed things up, but I also seemed to remember reading somewhere that were other ways to express hash values, and anyway, I am always very careful before I tell a judge that he or she is wrong. So doing a little online research, I learned that there are indeed other ways to display hash values using different binary based number systems, typically the 32 base or 64 base number systems. Base32 is defined in IETF RFC 3548, as using the characters A-Z and 2-7. While Base64 is defined in IETF PEM RFC 1421 as using the characters A-Z, a-z, 0-9, / and +.

My Online Investigation of Base32 Hash Math
Led to a Shocking Discovery

Coming back to the Warren opinion, the hash values “H4V … UTI” are not hexadecimal, but they could be either Base 32 or Base 64. At this point, I did a little more online research about Base32 hash, and quickly found that there are many websites where you can locate music and videos to download based on their hash values. Almost right away, by simply using Google, I located a site where you can find media to download based upon their SHA1 Base32 value. It then took less than a minute to find the web page where the Base32 SHA-1 hash values were listed that began with “H4V.” That is how all of the media on the site was listed, in numerical order based upon the first three numbers of their Base32 hash values.

There were 83 entires on the webpage whose hash values began with H4V. The site included listings of music and videos ranging from Beethoven’s Symphony No. 9 to a video of Lee Trevano’s Golf Instruction. One video listing which was 11.1 MB in size had a disturbing title that suggested it could contain the kind of porn referenced in Warren. It was dated May 29, 2003. I clicked on its hash value button and saw that the full SHA-1 hash value for this video was H4VIBLSKAZ477WRTKH7IURE6NXEDCUTI.

When I saw that hash value, it shook me up. The first and last three values exactly matched the hash described in Warren: H4V … UTI. My academic investigation of the mathematical properties of hash had led me right to the smoking gun in Warren! I knew from my article, and the research of Bill Speros described in footnote 168, that this match of the first and last three values meant there was a 98.6% probability that this was the exact same file referenced in Warren.  Mr. Warren was charged with a felony for distributing this same video. I think it is a crime to even have it on your computer.

I do not know for sure if it is the same file, since the Warren opinion nowhere states the full hash value, but in view of the description of this video, it is just too much of a coincidence for it not to be.  It was astonishing on many levels to see just how quickly you can find a file like this on the Internet, simply by knowing the first three hash numbers. 

It is probably not possible to actually download or view the file from this website. I do not really know for sure, since that would involve clicking on this file, which I was not about to do. But when I clicked on the link for Beethoven’s Symphony No. 9, a piece of media which I do not find morally reprehensible, it took me to another web page. This page had links to other computers where you may in fact have been able to download Beethoven’s music. (I did not try, recognizing that might be a copyright violation.) At that point, the referring website included a statement that it “ONLY HAS INFO ABOUT FILES, AND DOES NOT OFFER ANY FILES FOR DOWNLOAD.” Still, if any law enforcement agency wants to contact me for the full website address, including Cuomo’s group, I would be happy to provide it. It is really very easy to find, and so I assume the proper authorities are already well aware of this site and its hash values, or lack thereof. I am certainly no police officer, and even if I was, I would not have the stomach for this kind of investigative work. Reading the email of parties in civil suits is about as horrid as I can handle.

Judge Noce Was Right

This little investigation proved to me that Judge Noce and the St. Louis police were correct. There is a SHA-1 hash that has 32 places, not 40, and it can use the whole alphabet, not just A-F.

The hash value H4V … UTI is indeed a correct first and last place truncation of a full SHA-1 hash value. But it is a SHA-1 hash that is expressed in Base32, not hexadecimal. Although the hash values used in e-discovery are almost always hexadecimal, the hash values used in “Peer-to-Peer” websites include a variety of different numerical systems, frequently including the Base32 system.

In addition, in my brief investigation of the P2P webs, I learned that countless P2P type websites now commonly use the first three places of hash values as a convenient shorthand naming system. For all I know, the “perps” may also. As Judge Noce says, it is the convenient thing to do. So when will the e-discovery vendors start doing so too?

The “Wicked Quadrants” - Thoughts on a Possible Theoretical Construct to Understand Unethical Behavior in e-Discovery

The Wicked Quadrants

Electronic discovery seems to have more published decisions with judges complaining of attorney misconduct than any other area of law. The unethical behavior ranges from intentional fraud, to gross negligence, to simple attorney negligence. Yes, even occasional unintentional negligence is, in theory at least, unethical conduct. See Eg. Rule 1.1., ABA Model Rules of Professional Conduct:

A lawyer shall provide competent representation to a client. Competent representation requires the legal knowledge, skill, thoroughness and preparation reasonably necessary for the representation.

Some judges I know think that that attorney incompetence in e-discovery is so widespread as to present an ethical crises for the whole profession. Moreover case law, exemplified by Qualcomm, suggests there is far more to the sanctions being imposed by judges all over the country than “just” lawyer incompetence. When I began my career in 1980, the imposition of sanctions, especially against attorneys, was a very rare event, and motions based on spoliation were unheard of. Now they are common place. Why is this? It is a difficult and puzzling question.

Surely the profession has not suddenly become more sinister than before, although some suggest that the dominance of large firms as mega-business enterprises is causing a significant decline in overall ethics. Galanter, Henderson, The Elastic Tournament: A Second Transformation of the Big Law Firm, 60 Stan.L.Rev. 1867 (April 2008). There may be some truth to this observation concerning the overall decline of standards, but that would not explain why e-discovery, over all other areas of law, is so rife with malfeasance.

Part of the answer has to lie with the incredible advances in technology that we have seen in our lifetimes. Society has transformed from a nation-based industrial world, to a global-based computer world. The rapidity of this change in civilization is unprecedented in human history. The transformation has had profound effects on the nature and quality of evidence, and the process to find this evidence. See: Information Inflation: Can The Legal System Adapt? 13 Rich J.L. & Tech 10 (2007). In fact, e-discovery is born out of this paradigm shift in the law. See: Intellectual Foundation of Electronic Discovery.

Business and all other sectors of society have undergone the same rapid transformation. Yet they seem to be rising to the challenge of new technologies better than the legal profession. True, there have been some spectacular ethical disasters in business, symbolized by the collapse of Enron and Arthur Anderson. But once again, you could point the blame on their attorneys, especially their in-house counsel, who failed to guide these companies and keep them within the bounds of legal conduct. 

Most people I have talked with about this problem agree that the failure of the legal profession to keep up with technology can be blamed on two things: (1) the personality and intelligence type of most lawyers; and, (2) the failure of law schools to even try to adapt. Most lawyers are not strong in math, science, or engineering. There are exceptions, of course; we call them IP lawyers. But for the most part “The Law” attracts people who are gifted with a particular kind of liberal arts logic intelligence that inclines them to “computer-phobia.” Law schools have not done anything to counteract this trend. They are, after all, staffed with similar brilliant, but technologically challenged types.  Most have ignored the problem of e-discovery altogether, and do not even try to teach the subject. There are a few notable exceptions, such as Cumberland’s Law School with Judge John Carroll, and the University of Florida’s Law School with Holland & Knight’s Bill Hamilton. They are, however, the rare exception, and for the most part, law schools have not stepped up to the plate to address this problem.

Since the cause of lawyer computer-phobia is well known, the answer is also clear: law schools need to include electronic discovery in their standard curricula and broaden their recruitment and admission standards to include the technologically gifted.

The assault of technology on the law is a strong driving force behind the decline of ethics. That much seems clear. But this does not, in and of itself, provide a theoretical construct to understand unethical conduct in e-discovery. That requires a deeper probing into the rules of ethics and observation of legal practice. I am just beginning to think about this interesting problem. I hope to develop a full blown thesis to present at an upcoming symposium on e-discovery ethics at Mercer Law School in November entitled ”Ethics and Professionalism in the Digital Age.”

This will be the first academic event, that I know of, to seriously address issues of e-discovery ethics. The Key Note speech will be provided by Professor Monroe H. Freeman, the country’s leading scholar on legal ethics. There will be two panels. The first will be led by Jason Baron who will present “E-Discovery and The Problem of Asymmetric Knowledge: Some Thoughts on the Ethics of Search and Information Retrieval In Light of Recent Case Developments.” His panel will include Judge John Facciola. I will lead the next panel and present on “The Wicked Quadrants: A Theoretical Construct for Understanding Unethical Conduct in e-Discovery.” My panel will include Bill Hamilton. An edited transcript of the proceedings with be published in the 2009 Spring Edition of the Mercer Law Review. 

I have come up with a four-fold theoretical framework for analysis of unethical behavior in e-discovery, my so-called “Wicked Quadrants.” It includes the radical transformation of society as one of four factors to consider. I can summarize the four as: (1) technology challenge; (2) over zealous client representation; (3) stunted development of professional duties as an advocate; and, (4) legal incompetence.

I usually employ visual aids to assist in an analysis project as difficult as this, and so have prepared a series of charts to help organize my ideas. I am still considering whether these four factors are sufficiently complex for a full analysis of the problem. I am also still not sure I have included all of the important governing rule of ethics that have a strong impact on e-discovery. (After you watch the video presentation below, please let me know in a comment, or private email, if you think I have missed anything.) I am also still considering how the four criteria interact with each other to explain the varying types of unethical behavior that I see everyday in my legal practice and study of e-discovery case law. I already have a few thoughts on this, but they are not sufficiently developed at this time to share publicly. By November it should all come together. At that time I will write about it here and provide citations to the Mercer Law Review.

In the meantime, I have prepared a slide show that explains the basic concepts of the WICKED QUADRANTS. It includes quotes of the relevant ABA Rules, my charts, and graphics. I hope you enjoy the spotlight animations I made to present these slides in a QuickTime movie. Press the play button and wait a few seconds for it to start. I also include a link to the flat file PDF version of the video where the words are much easier to read, and in case you cannot view the video.

done

Here is the link to the The Wicked Quadrant PDF.

More “Must Read” 2008 Cases - Part Three - Including My Favorite, the Wesley Snipes Tax Evasion “e-Document Dump” Case

This is the third and final installment of the”Must Read” 2008 Cases series that I began two weeks ago. The series reviews, in alphabetical order, thirty 2008 e-discovery cases. They are good reference cases that have some lessons to teach, but have not previously been featured in this blog. Part One covered ten cases A-J in just over 4,700 words. Part Two covered seven more cases in 4,150 words, but I only got through two letters in the alphabet, H-I. This last stage will cover the rest of the alphabet, covering 13 more cases in 5,100 words. My favorite is the Wesley Snipes tax evasion case, which not many people know has an interesting e-discovery component to it. U.S. v. Snipes, 2007 WL 5041892 (M.D. Fla. Dec. 24, 2007). Of course, the Ninth Circuit case about the swat team officer in California who constantly used a police pager to send sexy text messages at work is pretty good too. Quan v. Arch Wireless, _F.3d_, 2008 WL 2440559 (9th Cir., June 18, 2008).

Mikron Industries Inc. v. Hurd Windows & Doors, Inc., 2008 WL 1805727 (W.D. Wash. Apr. 21, 2008).
This is an interesting opinion by Seattle District Court Judge Robert S. Lasnik interpreting Rule 26(c) and 26(b)(2)(B). Defendants in this breach of contract case sought an order shifting the costs of producing ESI to Plaintiff, arguing that the requests were burdensome and cumulative. Judge Lasnik denied the motion on both procedural and substantive grounds.

He denied the motion on the procedural grounds because Defendants failed to comply with the good faith meet and confer obligations under Federal Rule of Civil Procedure 26(c). Still, Judge Lasnik went ahead and addressed the merits of the costs shifting motion, and denied it again. He held that Defendants had not met their burden of proof under Rule 26(b)(2)(B). Defendants merely provided a cost estimate and described the electronic records as “inaccessible.” Here is the language used by the court, which begins by criticizing Defendant’s lackadaisical collection efforts:

Although defendants directed their employees to search their hard drives for responsive information, defendants have not demonstrated any search efforts beyond that limited inquiry. Responsive information may be discovered during a more thorough search of defendants’ non-backup ESI, including employee hard drives and active e-mail servers. Cost-shifting would not be appropriate in the context of this kind of search, as this ESI is considered reasonably accessible within the meaning of Fed.R.Civ.P. 26(b) (2)(C).

With regard to ESI located on defendants’ backup tapes, those courts that considered shifting the costs of electronic discovery to the requesting party were presented with more detailed information than that provided by the defendants in this case.FN2 In alleging that continued discovery of their ESI would be unduly burdensome, defendants offer little evidence beyond a cost estimate and conclusory characterizations of their ESI as “inaccessible.” Defendants have not provided the Court with details regarding, for example: (1) the number of back-up tapes to be searched; (2) the different methods defendants use to store electronic information; (3) defendants’ electronic document retention policies prior to retaining an outside consultant; (4) the extent to which the electronic information stored on back-up tapes overlaps with electronic information stored in more accessible formats; or (5) the extent to which the defendants have searched ESI that remains accessible.

This case is thus useful to show the kind of evidence that some courts require to support a cost shifting motion under Rule 26(b)(2)(B). It also shows, once again, that just telling your employees to find and collect relevant computer files, with nothing more, is not adequate. The opinion concluded with an order directing both sides to meet and have bona fide good faith discussions on e-discovery issues before coming to him with any more motions.

Peskoff v. Faber, 2008 WL 2649506 (D.D.C. July 7, 2008).
This is a decision by one of my favorite e-discovery magistrates, Judge John M. Facciola (shown right). He found that the producing party, the Defendant, failed to comply with discovery requests because there were significant gaps in the production. The gaps were explained, in part, by the fact Defendant “failed to deactivate network maintenance tools that automatically delete electronically stored information.” Id. at *4. Other omissions were never explained, due in part to the fact that Defendant failed to appear at the evidentiary hearing (never a good idea). This suggested intentional, or at least negligent, omissions in collection or production of relevant ESI.

As a result of Defendant’s failures, Plaintiff asked the court for permission to conduct a forensic examination of Defendant’s hard drive, and for Defendant to pay for it. Judge Facciola recognized that a forensic exam requests inaccessible information and thus made a full analysis under Rule 26(b)(2)(B). It provides a text book example on how the rule should be applied, except for a dicta statement that only the production of inaccessible data is subject to cost-shifting, citing to OpenTV v. Liberate Techs., 219 F.R.D. 474, 476 (N.D.Cal.2003). Id. at *2. In my opinion, that is only true if you are doing a 26(b)(2)(B) analysis to begin with. Otherwise, there are many other factual situations and other legal authority for shifting costs for accessible ESI. It certainly would not be wise to only employ cost shifting protection to requests for production of inaccessible data. See Eg.: Garrie & Armstrong, Electronic Discovery and the Challenge Posed by the Sarbanes-Oxley Act, 2005 UCLA JL&Tech 2.

Judge Facciola held that the requesting party, here the Plaintiff, had proven good cause under the rule to require production, and no cost shifting was appropriate. The $33,000 forensic exam expense must be born by the producing party. This was fair and equitable under the rules because there was $2.5 million in claims at issue, and the potential benefits of the discovery out weighed the costs. Further, Defendant’s failure to suspend the delete functions is what rendered much of the ESI at issue inaccessible to begin with. For these reasons Judge Facciola chose not to deviate from the “traditional rule that a responding party bears the costs of production.” Id. at *4. As Judge Facciola put it: “This is a problem of Mr. Faber’s own making and, consequently, the expense and burden of the forensic examination can hardly be described as ‘undue.’”

Quan v. Arch Wireless, _F.3d_, 2008 WL 2440559 (9th Cir., June 18, 2008).
This case involves a swat team police officer who sent many, many sexually explicit text messages to his wife during work hours. He used a police department pager intended for emergency communications. In spite of these facts, the Ninth Circuit held that a public employee has a reasonable expectation of privacy to text messages and emails, and found unlawful search and seizure by the employer. The police department had read the text messages on its pager in connection with an audit for text message over-charges, and this lead to disciplinary action against the swat team officer. The police department employer had relied upon its formal computer use policies and procedures, to which the officer had signed a written acceptance, to try to justify its actions. The policy clearly stated that there was no privacy for any electronic messages at work, including email and text messages. The Ninth Circuit indicated that it would have enforced these policies, but for the fact that the employee’s supervisor had implemented a different informal policy causing the swat team officer to have a reasonable expectation that his text messages would not be reviewed. The court held that the “operational reality” trumped the “formal written policies.” For this reason, the employer’s review of the employee’s text messages violated the employees privacy rights, and that of his wife.

The case is also important for its interpretation of the Stored Wire and Electronic Communications Act, 18 U.S.C. §§ 2701-2711 (1986). The Appeals Court held that a paging company was an “electronic communication service” governed by the act. Thus the company’s disclosure of the messages to the employer, who was the “subscriber” and not “an addressee or intended recipient of such communication,” violated the act.

Simon Property Group, Inc. v. Taubman Centers, Inc., 2008 WL 205250 (E.D. Mich. Jan. 24, 2008).
Magistrate Judge Mona K. Majzoub considered a motion to enforce a subpoena against a third party. The underlying action involved allegations of the Racketeer Influenced and Corrupt Organizations Act (RICO), various securities laws and other tort claims. The third-party argued that it should not be required to produce electronic documents responsive to the subpoena because it would be unduly burdensome and expensive. Their search of electronic files identified over 250,000 files. The requesting party offered to narrow the scope of the search by altering the time periods, search terms, and possibly dropping some computer servers from the search. The court enforced the subpoena as long as both parties agreed to narrow the scope in good faith because discovery of electronic records is contemplated under Federal Rule of Civil Procedure 45(d) and is common in business litigation.

Southern New England Telephone Co. v. Global NAPs, Inc., 2008 WL 2568567 (D. Conn. June 23, 2008).
You would be hard pressed to find a case with discovery violations more egregious than this one, which explains why the ultimate sanction of a default judgment was entered in the amount of $5,247,781, plus fees and costs of $645,760. Among the parade of horribles, Defendants failed to turn over their financial records in direct contravention of numerous discovery orders, lied to the court about not obtaining records from third parties, and willfully destroyed and withheld relevant documents. Additionally, Defendants erased computer data in bad faith by using disk wiping software called Window Washer and its “wash with bleach” option. This option overwrites files with random characters to make them unrecoverable. If that were not enough, Defendants then ran the Windows Disk Defragmenter program on the drive, making recovery even more impossible.

The court found Defendants had committed fraud upon the court and prejudiced Plaintiff to such a great extent that it would be unlikely that Plaintiff could ever prove its case. A default judgment under these circumstances was the only viable remedy. Here is how District Court Judge Janet C. Hall summed it up:

In conclusion, defendants’ behavior exemplifies the type of willful disregard for the process of discovery created by the Federal Rules of Civil Procedure that warrants the ultimate sanction of dismissal. Defendants “rolled the dice on the district court’s tolerance for deliberate obstruction,” and this court does not believe they should be allowed to “return to the table.” Bambu Sales, 58 F.3d at 853.

Square D Co. v. Scott Elec. Co., 2008 WL 2779067 (W.D. Pa. July 15, 2008).
Here is another case a defendant has managed to lose entirely by poor judgment in e-discovery. It is doubtful the merits of the case matter much any more here, since defendant and its counsel have lost all credibility with the court through their over-aggressive discovery tactics. They tried to evade discovery and then did not fulfill their responsibilities, nor follow the court’s order. The result is a costly forensic examination of all of their computers and sanctions that stop just short of a default judgment. Here are the words of District Court Judge Nora Barry Fischer, whose patience has obviously worn out.

In conclusion, while the Court has previously spoken of its frustration regarding the discovery process in this case, it is becoming clearer to this Court that Defendant Globe and its counsel bear much of the responsibility for the continual and unreasonable delays in effectuating the Court-ordered forensic inspection and for the other delays in the discovery process in this case. With that in mind, the Court wishes to impress upon Globe and its counsel that any further restrictions unilaterally imposed by it or its counsel on the forensic inspection (in any regard) as well as any other baseless barriers impeding the completion of discovery will be met with sanctions. The Court shall conduct a post-inspection status conference on August 19, 2008 at 10:00 a.m. and counsel shall appear in-person. The Court advises the parties that if additional discovery disputes arise, the Court may require them to utilize a special master, with costs to be borne by the parties.

Note here how Judge Fischer uses the threat of a special master and the expenses that will incur as a stick to try to cajole good behavior. This seems to be a tactic that more and more judges are using lately.

Sterle v. Elizabeth Arden, Inc., 2008 WL 961216 (D. Conn. Apr. 9, 2008).
This is yet another case where aggressive defense lawyers use old fashioned advocacy in e-discovery. As usual, they end up doing more harm than good. This is a wrongful employment termination case like Zubulake. Plaintiff requested the production of Department Stores Fragrance Group (DSFG) sales reports that tracked employee performance. Defendant claimed that seven key DSFG reports were no where to be found and so they could not produce them. Apparently the court did not believe Defendant’s convoluted excuses, and so ordered a forensic exam of Defendant’s computer system.

In summary, over the course of nine months, the Defense Attorneys’ position regarding the DSFG Reports went from: (1) not being able to produce the documents, to (2) claiming that the documents could not be found, to (3) offering to “recreate” the documents, to (4) claiming that producing the documents would be “overly burdensome,” to (5) hypothesizing that the Plaintiff fraudulently produced the June 2004 DSFG Report, to (6) after receiving Sachse’s incriminating email, producing four of the eleven missing DSFG Reports.

The court ordered a computer forensic expert agreeable to both sides to inspect Defendant’s hard drives to try to locate the seven missing DSFG reports. The parties agreed on an expert, but when he arrived at Elizabeth Arden, he was prohibited from taking any forensic images by Defendant’s attorneys, and denied access to certain parts of the computer system, including a network folder named DSFG, and the laptops of key employees likely to have the DSFG reports.

Defendant then filed a blustery motion for protective order, claiming that the expert had gone too far, and trying to prohibit further inspection of its computer systems. They even sought to have Plaintiff pay for Defendant’s attorney fees. Plaintiff responded with a motion for contempt based on Defendant’s violation of the court’s order, and a motion for sanctions and attorneys’ fees. The court granted Plaintiff’s motion for sanctions finding that defense counsel engaged in “obstructive tactics” evidencing bad faith and a disregard for the authority of the Court. Here are the words of District Court Judge Dominic J. Squatrito:

*10 Accordingly, although the Defense Attorneys may have provided some access to the Defendant’s servers and networks, the majority of the inspection was thwarted by the Defense Attorneys obstructive tactics. Such tactics are sanctionable because “[a] party that ignores or engages in delaying tactics, despite an explicit refusal to produce discovery, is still liable for sanctions pursuant to Rule 37.” Insurance Corp. of Ireland, Ltd. v. Compagnie des Bauxites de Guinee, 456 U.S. 694, 707-08 (1982) (finding that the trial court’s invocation of powers under Rule 37 was “clearly appropriate” based on “continued delay and an obvious disregard of its order” even though “petitioners repeatedly agreed to comply with the discovery orders”).

A new inspection was ordered and Defendant was ordered to pay all costs and fees. As a practical matter, the credibility of the Defendant and its attorneys is now shot. Their discovery gamesmanship has backfired terribly. At this point, I would guess their odds of prevailing to be about as good as UBS Warburg in Zubulake, i.e. - slim to none. Here is Judge Squatrito’s stern warning:

In addition to the costs and attorneys’ fees already awarded by the Court, the Plaintiff Attorney requests that the Court enter a default judgment. This request is denied, but not without warning to the Defense Attorneys. In the event that the Defense Attorneys do not comply in full with the directives contained in this decision, they may be found to be in defiance of these orders and to be disinterested in the resolution of this case on the merits. In such an instance this Court may strike the Defendant’s answer and enter a default judgment against it.

Treppel v. Biovail Corp., 2008 WL 866594 (S.D.N.Y., April 2, 2008).
This is another case where an over-aggressive refusal to cooperate unnecessarily drives up the cost of e-discovery. Defense counsel makes what appears to be a fairly forthright proposal to find relevant ESI and respond to pending e-discovery requests. He suggests that the parties reach an agreement as to which employees’ files should be searched and what search terms to use. Defense counsel proposed the following search terms and invited discussion: (i) Treppel, (ii) Jerry, (iii) Bank of America, (iv) Banc of America, (v) BAS, and (vi) BofA. (Personally, I do not think “Jerry” is a great search term, since it is a fairly common name, but that is beside the point.)

You would expect Plaintiff’s counsel at this point to respond by suggesting a few additional keywords, or making comments like I did as to the efficacy of the proposed search terms. You would expect cooperation and communication to begin to resolve the common problem of retrieval of responsive ESI from a large, complex computer system. But no, not in this case where there is a much higher level of virulence between the parties than usuall, and, it appears, their attorneys too. No, what happens here is Plaintiff’s counsel responds by stating that “it is defendants’ obligation to simply search its [sic] records and respond to those demands. Plaintiff has no obligation to assist defendants in the process by providing search terms or any other guidance.” Id. at *3. Then, after some discussions but no communication, Plaintiff responded with several discovery motions.

Magistrate Judge James C. Francis, IV, responded by denying Plaintiff’s request for a preservation order as premature, and ordered Defendants to “promptly conduct a diligent search, explain the search protocol they use, and produce the responsive documents so located.” Id. Defendants then proceeded with the search protocol and terms it originally proposed. Defendants searched the computers of 14 witnesses and multiple shared drives areas. For reasons not explained, Defendants also voluntary searched the backup tapes of two servers.

After the search, Plaintiff now wanted to get in on the act, and demanded that 30 new search terms be added, along with numerous additional ESI custodians, plus a search of several more backup tapes. No explanation as to why Plaintiff did not make any of these requests before, when the search terms negotiations were first proposed, and before Defendants carried out the court ordered search. Of course, I have my theories, that this is just old fashioned adversarial churning, tit for tat attempts at entrapment, not a bona fide search for truth. But that’s just a theory. Anyway, Defendants naturally refused to run yet another search with these new terms, stating the obvious, that it was too late and the request was too broad. The plaintiff then filed a motion to compel the search it wanted, and for sanctions based on Defendants inadequate preservation of evidence. These motions are the subject of this opinion.

I know how I would have ruled on Plaintiff’s motions, especially the motion to compel. If nothing else, to send a clear message about intolerance of e-discovery gamesmanship. But this Magistrate saw things slightly differently, and granted one part of the motion to compel. He ordered the restoration and search of two additional backup tapes, even though Plaintiff had apparently not specifically sought discovery from these tapes before the search and first order. At least the Magistrate did not require the use of the 30 new search terms Plaintiff wanted, nor the new custodians. Still, in my opinion, the search should not have been broadened in any way. Plaintiff had a chance to communicate and cooperate, and instead, Plaintiff’s counsel said, in effect, “not my job.” (Actual words: “Plaintiff has no obligation to assist defendants in the process by providing search terms or any other guidance.”) I am sorry, but as long a judges keep doing this, and rewarding one party’s refusal to cooperate, even if only in part, the bench only serves to encourage the kind of behavior they profess to oppose.

As to Plaintiff’s motion for sanctions, the Plaintiff sought an adverse inference instruction, and pointed to multiple oversights in preservation. The Magistrate correctly noted that Defendant’s preservation efforts were inadequate. At first, all Defendants did was provide verbal instructions to two key witnesses to start preserving evidence. There was no follow up, and apparently nothing at all was done during the first seven months of the case. The Court quite correctly relied upon Judge Scheindlin in Zubulake to find a dereliction of duty in this inaction. See Zubulake v. UBS Warburg LLC, 229 F.R.D. 422, 432 (S.D.N.Y.2004) (“Zubulake V” ) (“[I]t is not sufficient to notify all employees of a litigation hold and expect that the party will then retain and produce all relevant information. Counsel must take affirmative steps to monitor compliance so that all sources of discoverable information are identified and searched.”)

But the court went on to ignore Judge Scheindlin on another point and criticized Defendants for not preserving some of their monthly backup tapes. The court correctly quoted Judge Scheindlin “As a general rule, [ ] a party need not preserve all backup tapes even when it reasonably anticipates litigation.” Zubulake IV, 220 F.R.D. at 217. But then went on to incorrectly state that the law had since changed. For this dubious proposition, the Court cited only to Toussie v. County of Suffolk, 2007 WL 4565160, at *8 (E.D.N.Y. Dec. 21, 2007). Id. at *8. But if you read Toussie, you see that it incorrectly cites to Zubulake to support the position that “the law is now clear that any backup tapes containing the documents of a key player must be preserved and accessible.” This is how bad law gets started, much to the joy of some vendors. In any event, this incorrect statement on backup tapes was just a dicta passing statement in Toussie, and not at all essential to its holding. I have previously written a detailed blog on Toussie analyzing the unusual facts of that case. The Toussie court was justified under those circumstances to order a backup tape search in that case, and there was no need to turn Zubulake IV on its head to do that. In so far as Toussie and Treppel are cited for this proposition, they are creating bad law. There needs to be a good reason to preserve backup tapes, and in most cases (but not all) there is no need, because the relevant ESI will be found on live data.

Although I disagree in part with the sanctions road taken here, the end result of the Treppel court’s ruling on sanctions was just. Plaintiff’s request for an adverse inference was denied, and only an inspection of one suspicious laptop was ordered. All other sanctions were denied. In my view, more severe sanctions probably would have been awarded, if Plaintiff had not shot his credibility with the obvious gamesmanship in search.

U.S. v. Johnson, 2008 WL 2060597 (E.D. Va. May 15, 2008).
This is an interesting case to read for its facts, involving the criminal prosecution of Junior Johnson, the founder and CEO of now defunct, PurchasePro.com. This company was a dot com bubble that eventually burst. When it all started falling apart, Junior stopped at nothing, including the law, to try and keep his stock price afloat. For a while he was on top, making millions with a dubious business model involving business to business commerce and a virtual marketplace. At the end of 2000, the market capitalization of his company was almost $1 billion, and Junior’s own shares were worth over $236 million!

If you enjoy reading about how con men operate as much as I do (which is why I loved working on the Lou Pearlman case), you will get a kick, and several laughs, out of reading this 49 page opinion. It lays out an intricate trail of fraud, intrigue and audacious actions, some of which worked for Junior, at least for a while. In the end though he was convicted of numerous felonies, including “attempting to obstruct an official proceeding-his first trial-by giving his counsel an altered e-mail to use as evidence.” That is just one of many of Junior’s “boyish antics” that the opinion recounts in detail, and the only real e-discovery component to the case.

Junior’s first trial ended in a mistrial after his defense counsel withdrew upon learning that Junior had given him four altered emails. The attorney had been tricked by his client into placing altered emails into evidence, and using them in cross examination. This criminal defense counsel did absolutely the right thing upon discovering what happened. Eventually, the true story behind the altered email came out, and in Junior’s second trial, which this time included obstruction of justice, Junior now admitted to altering the emails. He claimed he did that just to see if his attorneys were actually reading all of the stuff, because it was obviously false, and surely his attorney would notice that if he took the time to read it. Id. at *39. He said he played this little prank on his counsel to prove his lack of attention to detail.

The court did not buy this, and instead held that it was “easily foreseeable that giving your trial counsel an altered piece of evidence in the midst of trial naturally and probably will interfere with the trial.” The court found that Defendant had the requisite criminal intent and found him guilty of obstructing justice. Here is the governing law:

To convict Junior of obstruction, the Court must find, that Junior “corruptly … obstruct[ed], influence[d], or impede[d] any official proceeding, or attempt[ed] to do so.”18 U.S.C. § 1512(c)(2). As explained earlier, the word “corruptly” connotes wrongfulness or impropriety. Arthur Andersen, 544 U.S. at 705;see United States v. Matthews, 505 F.3d 698, 706 (7th Cir.2007) (employing Arthur Andersen definition of “corruptly” in the context of section 1512(c)).

I point out that this law on its face applies to any “official proceeding,” not just criminal trials. There is a lesson here for all litigants, including those in civil law suits. Do not try to sneak an altered electronic document into evidence. It may seem easy, and you might even get away with it some of the time, but it is fraud and, if detected, you could face harsh criminal penalties.

U.S. v. Snipes, 2007 WL 5041892 (M.D. Fla. Dec. 24, 2007).
This is the well-known tax evasion case against movie star Wesley Snipes. Many also consider this case to be a good example of an unfair e-document dump. See what you think.

Just before trial in Ocala, Florida, the government produced an electronic database containing the equivalent of 1.6 Million pages of documents! Wesley Snipes and his co-defendants moved for a three-month continuance of trial to have time to sort through the database. They claimed that it was all newly discovered evidence. The government denied it was new evidence and claimed the defendants were previously provided an opportunity to inspect, but declined to do so.

Senior District Court Judge Terrell Hodges denied the request for continuance and concluded that defendants “had an ample opportunity to review and analyze the discovery material and prepare adequately for the trial of the case.” Judge Hodges explained that the ESI in the database consisted primarily of “computer software programs or copies of documents previously provided to the defendants in hard copy or other form.” Not sure exactly how they did that. The Court also seemed to be impressed that the government only intended to use 20 documents at trial that originated solely from the electronic form. But what about the 1,599,980 pages of documents they did not not want to use?

Wesley Snipes was acquitted of the tax evasion felony charges, even though he did not put on any defense witnesses. He was, however, convicted of three misdemeanors and sentenced to the maximum penalty of three years in prison. He has been allowed to remain out on bail, and even to travel overseas, so that he can continue to make movies such as GalloWWalker while his attorneys appeal the convictions to the Eleventh Circuit.

Waste Services, Inc. v. Waste Management, Inc., 2007 WL 1174116 (M.D.Fla. April 18, 2007).
This is one of the first cases in the country to consider what e-discovery costs can be awarded as a court cost to the prevailing party after a summary judgment. The decision of Orlando District Court Judge Anne C. Conway affirmed the recommendations of Magistrate David A. Baker. Judge Baker disallowed some of the costs for “tiffing” as excessive, but allowed other less expensive tiffing costs incurred with a different vendor. Interestingly, all of the costs of “blow back” (converting electronic documents to paper) were allowed.

WIREdata, Inc. v. Village of Sussex, 2008 WL 2512963 (Wis., June 25, 2008).
The Supreme Court of Wisconsin reversed a lower appeals court recently, much to the relief of state and local governments around the country. The intermediate appeals court had held that municipalities violate the Open Records Law when they deny citizens access to a government database, and only provide copies of the records in PDF format. WIREdata, Inc. v. Village of Sussex, 298 Wis.2d 743, 729 N.W.2d 757 (Wis. App., 2007). The court had stated, “The organization and compilation of the data into the Microsoft Access database, done at public expense, allows greater ease of public access to the public assessment information. In keeping with the letter and spirit of the open records law, we will not allow the municipalities to deny [Plaintiff], and others who seek the information, the value-added benefit of this computerization.”

Allowing citizens access to live databases poses a host of technical and other problems to governments trying to comply with Sunshine laws. The Wisconsin Supreme Court realized this and reversed. It is all explained in detail in a 30 page opinion, that only a Foley lawyer would want to read in full. Here is the key section of the ruling:

We share the DOJ’s concern, as expressed in its amicus brief, that allowing requesters such direct access to the electronic databases of an authority would pose substantial risks. For example, confidential data that is not subject to disclosure under the open records law might be viewed or copied. Also, the authority’s database might be damaged, either inadvertently or intentionally. We are satisfied that it is sufficient for the purposes of the open records law for an authority, as here, to provide a copy of the relevant data in an appropriate format

Xpel Technologies Corp. v. Am. Filter Film Distribs., 2008 WL 744837 (W.D. Tex. Mar. 17, 2008).
In this copyright case, the magistrate judge granted plaintiff’s motion for an expedited forensic examination of Defendants’ computers. The court stated that good cause was shown at a hearing, but did not explain what it was. The party moving for the examination (Plaintiffs) were required to pay for the exam. The court set out a specific protocol for the computer expert and the parties to follow in the forensic examination of the computers. Care was taken to preserve the confidentiality of Defendants’ information. The protocol included a hashing requirement: “All images and copies of images shall be authenticated by generating an MD5 hash value verification for comparison to the original hard drive.”

The Days of the Bates Stamp Are Numbered

As a kind of strange lawyer-mid-life-crisis, I wrote my first law review article last year: HASH: The New Bates Stamp, 12 Journal of Technology Law & Policy 1 (June 2007). Following tradition, I tried to make the opening sentences as clever as possible:

For over one hundred years, complex litigation has relied upon the ubiquitous Bates stamp to try and maintain order and clarity in paper evidence by placing sequential numbers on documents. In today’s world of vast quantities of electronic documents, the days of the Bates stamp are numbered. Instead, the future belongs to a new technology, a computer-based mathematical process known as “hash.” (emphasis added)

Ok, maybe not so clever, but still, I was delighted to see an article this week entitled Bates Stamps’ Days May Be Numbered by Tom O’Connor in Law.com’s Legal Technology section. No big surprise here as I met Tom a few weeks ago, and we talked about hash. (I tend to do that, a lot.) I liked how Tom saw the conversion from Bates stamping to hash as symbolic of a paradigm shift, not only in e-discovery, but in the world at large. Tom and a few others, such as Craig Ball, see a significance in the move to hash beyond what I understood when I wrote the article. They also have a better grasp of how this fits with other e-discovery technologies and procedures to facilitate what Tom claims are huge savings in time and money. I gave Tom a copy of my article, as he had heard about it from Craig but not yet read it. (Yes, I usually keep an extra copy in my briefcase.)

I mentioned Tom’s ideas in a prior blog, e-Discovery at the Harvard Club in New York City, based on his presentation at the CLE. The article Tom has since written, Bates Stamps’ Days May Be Numbered, provides more meat for the bones, which I will attempt to summarize here and place into proper hash context. For still more information listen to Monica Bay’s recent interview of Tom on Legal Talk Network.

For those not real clear on what hash is, and what it could possibly have to do with the 19th Century Bates stamp shown above, I suggest you read my law review article. But if the thought of reading a 44 page academic paper with 174 footnotes leaves you cold, I suggest you try my Hash Page summary instead, or my earlier blog on Hash. They will give you a pretty good idea of how hash is the mathematical foundation of e-discovery, not a corned beef dish, and why this math should render sequential numbering obsolete. There are also many interesting comments left on these blogs by experts in the field, including an esoteric argument I had with a few vendors concerning the legal efficacy of hash in ESI authentication. These short articles do not go into law-review-depth, but do lay a helpful predicate to understand what Tom is talking about.

Tom’s article begins by noting that most people doing e-discovery today still rely on Bates stamping. They scan and sequentially number ESI as if it were a piece of paper. Then he observes, as I did in my introduction, that this system will not work “in today’s world of vast quantities of electronic documents.”

But that process is simply not effective when dealing with terabytes of data. To address the sheer volume, many vendors are advocating a new way of working with electronic documents that can reduce costs as much as 65 percent by eliminating the need for text extraction and imaging in the processing phase. Beyond immediate cost savings, this approach also provides cheaper native file production, reducing imaging costs for production sets and saving up to 90 percent of the time needed to process documents. How? By not using Bates numbers on every page.

Later Tom explains that the alternative to Bates numbers is hash values. But first, he details how and why this conversion can save so much time and money:

Currently, to provide Bates numbering, many vendors generate TIFF images from native files and then Bates number those images. But this process complicates native file review and — at anywhere from eight to 20 cents per TIFF — adds considerable cost to the process. Typically, during processing, data is culled, de-duplicated; metadata and text are extracted; and then a TIFF file is created. An unavoidable consequence is that the relationship of the pages to other pages, or attachments, is broken — and then must be re-created for the review process. Page-oriented programs handle this by using a load file to tie everything together from the key of a page number. But most new software use a relational database that stores the data about a document in multiple tables. To load single page TIFFs into a relational database involves a substantial amount of additional and duplicative work in the data load process.

These steps are avoided by changing to an identification system based on hash values of entire ESI files (which Tom here calls “documents”) that eliminates the need for tracking of individual pages. Here is how Tom explains it, using a lot of e-discovery oriented tech-talk, which, if he is speaking, is usually tempered by a few laughs and war stories:

A document-based data model, rather than a page-based approach, eliminates the text extraction and image creation steps from the processing stage and cuts the cost of that process in half. Documents become available in the review platform much faster — as imaging often accounts for as much as 90 percent of the time to process. This enables early case assessment without any processing, by simply dragging and dropping a native file or a PST straight into the application — which cannot be achieved with the page-based batch process. Relational databases allow for one-to-many and many-to-many relationships and support advanced features and functions — as well as compatibility with external engines for tasks such as de-duping and concept searching. Applications that support these functions — such as software from Equivio, Recommind and Vivisimo Inc. — are all document-based and will not perform in the old page environment. Programs that use the document model can eliminate batch transfer. This process (See Diagram 1 below) increases data storage due to the need for data replication in the transfer process and is also prone to a high rate of human error. And elimination of the time that inventory (in this case, electronic data) is stationary will eliminate overall cost as well as reduce production time

Tom’s diagram above shows the Bates stamp work flow model for traditional Tiff image e-discovery process and review. This procedure treats ESI as if it were paper, and uses sequential numbering, instead of hash, to identify information. According to Tom, this traditional procedure requires a number of time consuming and expensive batch transfer processes. He says these steps are unnecessary and can be eliminated in pure native review that relies on hash. The more simplified “Bates-free” process is shown by Tom’s diagram below. In his words, this is “an easier, faster and more cost-effective e-discovery process.”

Tom concludes that:

A modern litigation support program must be able to review native documents that are not just paper equivalents, and directly enable review of any file that is in common use in business today. The future belongs to these new technologies, where native files are processed without the need to convert to TIFF and are identified by their unique hash algorithm. Attorneys and clients who focus on a document-based system will save time and money and can conduct native file review. In today’s world of vast quantities of electronic documents, the days of the Bates stamp are numbered.

I could not agree more, especially since, unlike the tile, Tom now says the “days are numbered” and not “may be numbered.” I have no doubt about it, even though it may still take many years to get there. Old habits die hard, especially in the legal profession. Still, some day, Bates stamping will seem as quaint and antique as the original Bates numbering machine itself. The original shown above was invented in 1893. The first section of my law review article explains the history of this invention, and how Thomas Edison (shown right) purchased the patent from Edwin G. Bates. Then I go into the theory of hash and native ESI. I explain that hash is the digital fingerprint that identifies every electronic file, and reveals any change in the file. I also explain how hash is used in various e-discovery processes, and examine just about every legal decision ever written which mentions hash algorithms.

In case you have never seen a hash value before, here is an example: 4C37FC6257556E954E90755DEE5DB8CDA8D76710. There are many different types of hash formulas, but all produce lengthy alphanumerics hash values such as this. The two most popular are the SHA-1 hash algorithm which creates a 40 place hash value (shown above), and MD5 hash which produces a 32 place value. Both are too long for a practical naming convention to replace a Bates stamp. So I propose that the value be truncated and only the first and last three places be used. Thus the above hash would be shortened to 4C3.710 . I also propose that the # symbol stand for hash. (The # symbol is already commonly known as the hash mark in most of the world, but in many English speaking cultures, including the U.S., it is also called the number sign or the pound sign). So I propose to abbreviate the above SHA-1 hash with #4C3.710. Some of the technical details of this naming protocol are addressed in the law review article. Others will have to be worked out with time and experience, and the adoption of more standards in the e-discovery industry.

I conclude my article by imagining what a courtroom of the future might be like without the Bates stamp:

In countless courtrooms today, a mantra something like this is heard often: “I am handing the witness a document pre-marked as ‘Trial Exhibit 75’ and Bates stamped as ‘Dr. Smith 0573.’” In the future, the author expects something like this will be heard instead: “I am putting on screen for the witness to view an ESI file pre-marked as ‘Trial Exhibit 75’ and hash marked as ‘Dr. Smith Hash 4F7.C3B (Dr. Smith#4F7.C3B).’” The ESI file may still sometimes be converted to paper, in which case it could be handed to a witness, instead of put on a screen, but the same naming protocol would apply and it would bear a “hash mark” somewhere on the bottom: “Dr. Smith#4F7.C3B.”

Sorry, Mr. Bates, your one hundred-year-plus reign is over.

ABA Litigation Section Reacts to the Qualcomm Case and Recommends e-Discovery Checklists

The Litigation Section of the American Bar Association has published an online article on Qualcomm v. Broadcom. Written by Kristine L. Roberts, Litigation News Associate Editor, the article is significant for its glimpse into the thinking of ABA leaders on electronic discovery abuses. Essentially the ABA litigation leaders remind practitioners of the importance of discovery, and recommend e-discovery checklists as a good way to stay on top of the process and avoid another Qualcomm. While I agree that checklists can be useful, they have their limits, and in my view must be supplemented with expert advice, not to mention a strong sense of ethics and professional responsibility. 

Erica Calderas

Erica L. Calderas is the Co-chair of the Section of Litigation’s Pretrial Practice and Discovery Committee.  Erica is quoted in the article as saying:

The Qualcomm decision reminds all litigators—in a very forceful way—of the serious obligations we undertake in responding to discovery.

She is right on there. Discovery, especially complex e-discovery, is not something you can just delegate to a first year associate and forget about it. It is critical to the outcome of most cases, and can be easily messed up if not done right.

Erica recommends that attorneys use standard form e-discovery checklists in every case to make sure they cover all of the bases and avoid e-discovery violations. Good advice. This is especially important for a general litigator who does not have the assistance of an e-discovery specialist in a small case. Erica specifically recommends that attorneys:

[Use checklists to] ensure that you apply a consistent protocol in any new matter—for example, that you routinely instruct your client to preserve evidence, that you identify witnesses with knowledge, that you determine how the client maintains its documents, that you ask the right questions regarding where potential documents may be located, and that you ask about additional relevant documents and potential witnesses in every witness interview.

Checklists and Specialists

Law firms are now beginning to create and employ such checklists as a routine matter in all litigated matters. For instance, many already follow Erica’s advice and routinely instruct their clients on preservation duties at the beginning of a case. This is not a mere CYA exercise. For many clients, even otherwise very sophisticated ones, it can be a real wake up call. Many in-house counsel are, for instance, unaware of automatic ESI deletion programs, PC and backup tape recycling, forensic collection, and the like. They may need significant help to implement an effective preservation hold and collection program.

A few law firms are taking this a step further and recommending to their litigation attorneys that an e-discovery specialist be included in any significant case. This is not a hard and fast rule; merely a recommendation. In some firms, this advice is often not followed until after there is a problem, instead of at the commencement of the firm’s representation when their input could do the most good.

One large firm, Hunton & Williams, has gone beyond mere recommendation. They have promulgated a rule requiring e-discovery attorney involvement. In their words, they have begun “implementing requirements that an e-discovery specialist be assigned to all significant matters involving ESI.“ Hunton & Williams has 1,000 attorneys in 18 offices, and, of course, many e-discovery specialists. They are, to my knowledge, the first law firm to explicitly make this a requirement, not just a recommendation. Hunton’s Sherry Harris, whom I met last week at the Harvard Club CLE, brought this to my attention and obtained permission for me to share this. This is an important step and the management of Hunton should be congratulated. I expect that other law firms will follow in their footsteps, and eventually this will be commonplace. This is far more effective a solution than just distributing checklists to all litigators and hoping that everything gets done right.

The participation of e-discovery specialists can work seamlessly if the law firm requires it, and if the firm actually has such attorneys to carry it out. But at this point very few law firms actually  have specialists like that, and, of course they do not require what they cannot deliver. Today most law firms, especially small to medium size firms, do not have these specialists. They must look to outside entrepreneurs for assistance when there is a significant matter involving ESI. Of course, I am not saying that every litigated case needs that kind of input. The principles of proportionality and economics must always be followed, and many cases today still do not have a large e-discovery component. 

From what I have seen, although there are many e-discovery vendors, there are still only a few attorneys who specialize in e-discovery. Their numbers are, however, beginning to increase, especially among younger lawyers. The few who are full time e-discovery lawyers typically operate as independent entrepreneurs, or in small groups, or are employed by large vendors and consulting companies. This allows them to consult and be retained by other firms. A few e-discovery attorneys are shareholders, or of counsel, to some of the nation’s largest law firms, such as Hunton & Williams. Over half of the top 50 firms have e-discovery lawyers, but even then with varying levels of expertise. These big firm attorneys are usually fully occupied serving the litigators in their own firm, and are only rarely retained by other law firms as co-counsel.

Instead, the e-discovery lawyers who are on their own, or with consulting firms, are the specialists usually retained by law firms, both big and small, who lack attorneys with such arcane skills. As mentioned, they are usually called in to assist on projects after there is trouble of some kind. It is always challenging to bring in an outside attorney as an expert to assist in a case, but it is particularly difficult when it occurs after a problem develops. For one thing, how do you explain “the cleaner” to the client? No doubt it is the fault of the other side, or perhaps the judge. There can also be relationship issues when new attorneys from different firms work together for the first time. This is especially difficult when the trial attorney in charge has made a mistake and does not want to hear about it, nor understand the complexities involved. Yet, this is typically how and when most e-specialists get involved in litigation.

David Soley

Also quoted in Kristine Robert’s article was David A. Soley, of Bernstein Shur, Co-chair of the Section’s Trial Practice Committee:

We should not be surprised by the ruling, [the opinion] reflects what day-to-day practice ought to be. Attorneys are professionals and have professional standards to uphold, including a duty of good faith and reasonable inquiry in responding to discovery.

I assume David was referring to the sanctions imposed against Qualcomm and its attorneys for not upholding professional standards by trying to hide over 30,000 emails critical to the outcome of the case. David goes on to say that:

because lawyers will be held responsible for their clients’ production of documents, lawyers must go to the site where documents are kept. . . . the lawyer must understand what the client did and then verify it.

Once again, this comment (in my opinion) verifies the need for trial lawyers to obtain the assistance of e-discovery specialists in any large case involving complex computer systems. Counsel must not only understand what the client did, they must be sure their actions complied with the Rules and met the minimum forensic standards for admissibility as evidence. Also, they need to have the backbone to correct a client who screws up, or, as in the Qualcomm case, wants to hide the ball. 

David, who is himself a trail lawyer specializing in real estate litigation, does not talk about retaining e-discovery experts, but again suggests use of checklists.  Here is Kristine Roberts’ report of his comments:

Calderas recommends that to avoid e-discovery violations, attorneys should use checklists to “ensure that you apply a consistent protocol in any new matter—for example, that you routinely instruct your client to preserve evidence, that you identify witnesses with knowledge, that you determine how the client maintains its documents, that you ask the right questions regarding where potential documents may be located, and that you ask about additional relevant documents and potential witnesses in every witness interview.” Calderas also suggests that litigators enter into agreements with opposing counsel regarding what search terms will be used, the places at which relevant evidence may be found, and the persons whose files will be searched.

This is all good advice to be sure, but is it sufficient in a complex case involving large amounts of ESI?

ABA’s Checklist to Avoid Qualcomm’s Fate

The article concludes with a checklist summarizing the recommendations of the ABA leaders on how, as they put it, “to avoid Qualcomm’s fate.” 

1. Use checklists and develop a standard discovery protocol.

2. Understand how and where your client maintains paper files and electronic information, as well as your client’s business structures and practices.

3. Go to the location where information is actually maintained—do not rely entirely on the client to provide responsive materials to you.

4. Ensure you know what steps your client, colleagues, and staff have actually taken and confirm that their work has been done right.

5. Ask all witnesses about other potential witnesses and where and how evidence was maintained.

6. Use the right search terms to discover electronic information.

7. Bring your own IT staff to the client’s location and have them work with the client’s IT staff, employ e-discovery vendors, or both.

8. Consider entering into an agreement with opposing counsel to stipulate to the locations to be searched, the individuals whose computers and hard copy records are at issue, and the search terms to be used.

9. Err on the side of production.

10. Document all steps taken to comply with your discovery protocol.

Once again, all good advice, so long as you understand the limitations of such general advice. Further, if the point is to avoid another Qualcomm, mandatory ethical training should be included, along with the admonishment to walk away from any client who would have you hide evidence or lie to the court. There is no price on a sound night’s sleep. 

Limitations of Checklists

The above ten step checklist is, in my view, only helpful as a general starting point. Law firms should establish much more detailed forms and procedures to do e-discovery right. I know I have personally spent weeks doing just that. My firm, like a few others who have made such efforts, naturally holds such information as a trade secret. You can of course find many checklists online from a variety of sources, but they will all be generalized, and serve only as a starting point for further research, or teaser for retention of services. For one pretty good example, see the e-discovery checklist included in the Association of Corporate Counsel website. The same essentially holds true for form-books and commentaries. No complex area of law can be solved with simple forms and checklists, although again they can be helpful as a starting point. See for example, Electronic Discovery and Records Management Guide: Rules, Checklists and Forms 2008 ed. by Jay E. Grenig, Browning E. Marean and Mary P. Poteet, and Arkfeld’s Best Practices Guide for ESI Pretrial Discovery-Strategy and Tactics (2008-2009) by Michael Arkfeld.  This is especially true of e-discovery which is a combination of the fields of law and technology. In e-discovery, the facts are always different and rapid changes in technology quickly makes yesterday’s solution obsolete. 

Even if the detailed forms and checklists developed by a few experts for private use were no longer confidential, these checklists would not, by themselves, do that much good. They are meant to be used with the assistance of the experts who created them. Forms and checklists require background knowledge and team work with experts to function properly. They work best as a general guide, and reminder not to overlook necessary steps. They also let you know when and how to call for help. Sometimes just knowing what you do not know is half the battle. Step 7 in the above ABA checklist recognizes this in recommending employment of an e-discovery vendor. But be careful in relying too much on some vendors, especially those who are little more than copy-shops and have no in-house legal input.

The truth is, without experience and occasional guidance, simple checklists alone can be counter-productive. They can easily be misunderstood and provide a false sense of confidence. Sometimes it pays to be a little worried and concerned. I am sure that is one of the lessons Qualcomm’s former lawyers have learned. Perhaps the great poet Alexander Pope, whom I have quoted before, said it best in his An Essay on Criticism (1709): 

A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again.

 

e-Discovery at the Harvard Club in New York City

The stately 19th Century Harvard Club this week hosted a cutting edge 21st Century conference on e-Discovery. It was organized by ALM, who also puts on the Legal Tech trade shows. I was curious to see this private club and happy to speak at this event for N.Y. paralegals and attorneys. I was not disappointed by the wood-paneled Harvard atmosphere (where I took a few photos shared here), nor the content of the CLE. The event entitled Managing Today’s Discovery Process was well run, thanks to the organizer, Karen Abrams of ALM, and the program chair, Sherry Harris, the Senior Case Management Specialist for Hunton & Williams.

David Shonka of the FTC

Before the event, I got a chance to speak with the head of the Federal Trade Commission’s e-discovery team, David Shonka. David is also the FTC Assistant General Counsel for Litigation. David, like me, didn’t go to Harvard and was delighted to see the inside of the Harvard Club for the first time. We started talking about Qualcomm and both agreed it was The Big Case of the year. He endured my righteous rant about Qualcomm and attorneys who do not play by the rules, the ones who deliberately hide evidence they know should be produced. We also agreed that this kind of unethical conduct was nothing new, nor unique to e-discovery. It has always gone on, but in the olden days of paper discovery, it was far harder to expose. Today, with electronic discovery, hide the ball is suddenly a high risk undertaking. David picked up on my somewhat angry tone (over the years I have suspected several opposing counsel of playing this game, but have never been able to prove it). He said that in view of my strong opinions on this topic, I would probably like his presentation.

He was right. The FTC e-discovery team leader began with an overview of the incredible facts of the Qualcomm case. This was a prelude for this primary message, that there are only three fundamental principles to follow in e-discovery. They are, in his words:

Don’t Lie.
Don’t Hide Things.
Don’t Make Promises You Can’t Keep.

David pointed out that a law firm’s reputation for truth and honesty are key. If David thinks he is dealing with a lawyer that does not follow these fundamental precepts, then the FTC will naturally be much more demanding in their requests for information, and harsh in their treatment. Conversely, David is willing to negotiate and exercise leniency when an attorney is honest and forthcoming, and reveals the bad with the good. This attitude, in my experience, is also followed by most judges.

On the point of “don’t hide things,” he stated that anytime you have to have meetings, to try to “figure out an argument why something is not responsive, it is responsive.” You should produce it, you should not try to hide it. If not, in David’s words, you risk the principle that: “You are known by your last lie.” And that reputation will last for a long time. No doubt the boys in the 18th Century Harvard Philosophy and Advocacy Clubs (certificates shown above) would have agreed.

The FTC e-discovery chair stated that, in his view, parties have three general options for responding to discovery:

1. Hardball - prolongs investigation and is costly.
2. Let’s-see-if-they-can-find-it - again, prolongs investigation and is costly.
3. Co-operation - fastest, surest results and is the least costly.

David said the third path of “negotiation” is only the way to go for all e-discovery situations, but especially if you are responding to the FTC. No doubt the Harvard class of 1925 shown below would agree. (Note the sign they are holding on the right is not about one of my ancestors, instead it says “The Mess is Still Fundamentally LOUSY”, which is, I think, a reference to the food.)

Aside from general principles, David also had some practical advice on e-discovery, and how to find the relevant information, and not waste your time on non-responsive ESI. Here is his chart of the process he recommends:

Process of Narrowing ESI:

1. Preliminarily Identify Issues and Sources of Information:

a. Key players (12-36 max in any case)
b. Occasional players (only search if you have to)
c. Fringe players (rarely ever search)

2. Data

a. Enterprise systems
b. Local systems (particular office or groups)
c. Individual systems
d. Archives
e. Back-ups (and Legacy Data)

Most of the time all of the relevant data needed for a case will be stored on the key players’ Enterprise, Local and Individual systems. Sometimes you may also need to look at Archives too, depending on what you find in the more easily accessible stores, and how difficult it is to get at the ESI on Archives. Back-up tapes and Legacy Data are not usually needed. David explained that the FTC typically only requires two daily backup tapes be preserved, just in case they want to look at them later, which they usually don’t. He noted with a chuckle that the FTC picks which two tapes to preserve, not the respondent, and they usually just pick two at random.

On the issue of Litigation Holds, the FTC team leader recommended that a senior, experienced attorney personally supervise the preservation process and deal with all of the key players directly. This is at odds with the practice of many companies and firms that tend to delegate this task to younger, less experienced attorneys or paralegals. Probably not a good idea to do that when dealing with the FTC. If a second year associate messes up, you are not likely to have much sympathy. If an older guy like me screws up (yes, it happens), maybe they will cut you some slack.

On a promising note for large companies, he said that parties and the courts should always remember that companies are not in the business of preserving and holding evidence to produce to adversaries who may someday sue them. If e-discovery becomes so burdensome that it impairs businesses’ operations, then something is fundamentally wrong.

David’s presentation also included the fundamental message of this blog, that building an interdisciplinary team is key to e-discovery compliance. He recommended, as do I, that the team include IT, in-house and outside counsel, and vendors. He also advised that you “get everybody in the same room.” Otherwise, you will inevitably play the old child’s game of telephone, where a simple message is whispered for one person to the next, and by the time it reaches the last person, it is totally screwed up. He said it may seem expensive to some companies to assemble such a multidisciplinary team, and have them meet regularly and in-person, but he is convinced you save money in the long run.

Apparently the FTC has been using such a team approach to e-discovery for several years now and they are pleased with the results. He told me before the CLE started that they now have two attorneys who only do e-discovery, and they have help from a number of techs and paralegals. Of course, the FTC cannot force companies under investigation to also use a team approach, and specialized attorneys, but David did say publicly that they will sometimes refuse to meet with a company’s lawyers unless they bring the IT liaison with them to the meeting.

David also spoke of the serious risk of just relying on custodians for self-collection. They may print out, or transfer to a disk, but they are likely to do it in a way that messes up the metadata. He stated that metadata is only rarely needed for production, and depends on the case, but you should still try and preserve it as best you can. Still, the main reason you should not rely on custodians alone for collection is that they are “self-interested.” They may, for instance, want to avoid embarrassment and not produce certain very relevant emails that they wished they had not written. In his opinion, you can do the collection in-house, and do not have to hire an outside vendor, but you should use a qualified technician to go to the computers and collect the data, and not just rely on the custodians. As to forensic imaging, where outside experts are typically used, that is only rarely needed in special cases where there are indications of criminal conduct.

Sherry Harris

Sherry provided the opening keynote address. She has over 30 years experience with Hunton & Williams and is now the dynamo behind their e-discovery efforts. Although not an attorney, she knows far more about the subject than most experts. She commented on my sports analogy blog on e-discovery, and said how important it was for paralegals to help make sure the attorneys they support never drop the ball. Like most everyone else, she also talked about Qualcomm, and the outside counsel who now stand to be sanctioned. She emphasized that this is something you never want to have happen to your law firm. One way to avoid this is for lawyers and paralegals alike to, in her words, “stay aware of evolving case law.” She also said that in her experience, “traditional discovery is gone forever; but right now people are resisting the changes.” Her advice: “Make technology your friend.”

Vendor Selection and Negotiation

Next, Jennifer Tomaino, attorney member of the Verizon e-Discovery Team, and Oliver Gierke, Litigation Case Manager for White & Case, presented on Vendor Selection and Negotiation. It would be hard to imagine two people better qualified to speak on that topic. Jennifer is Patrick Oot’s “right hand man” (so to speak) for Verizon, and so has a good perspective on corporate e-discovery teams and their procurement of vendor services. Oliver is a key technical member of the White & Case e-discovery team, a law firm with 2,300 lawyers in 37 offices located in 25 countries.

Jennifer and Oliver both noted that paralegals play a key role in vendor selection because they work closely with vendors to get the work done, and are in the best position to evaluate performance. They also both commented on how big the e-discovery industry has become, now approaching three billion dollars a year

Jennifer Tomaino stated that Verizon does its own preservation and collection, and vendors do not usually get involved until the procession stage. Verizon is looking to take even more of the e-discovery process in-house in the next few years, and may move into review. In her experience, the review tools that the vendors offer are all fairly similar. She cautioned that “technology can help, but is only as good as the people using it.”

Jennifer suggested that you look for recommendations from others who have recently used a vendor. Also, you may want to employ a formal two-step procedure where you issue an RFI - request for info, and then an RFP - request for product proposal. She suggested you study the Sedona’s sample RFP in their vendor selection paper, but do not simply use the form without thought and customization. Jennifer also noted, and Oliver agreed, that a Vendors’ project managers are key. They need to provide quality, accessibility and responsiveness (24/7). Since Verizon is, in Jennifer’s words, a serial litigant, they can get the best rates because of the volume. Still, try and be creative in making deals with vendors, who usually charge b