The Latest Sedona Guidelines


Sedona Arizona in April2007

The Sedona Conference is the undisputed thought leader in the field of e-Discovery. Here are Ralph Losey's personal reviews and comments on its new publications on search and retreival, the fundamental principles of e-discovery, litigation holds and email management.

Sedona's Commentary on Search, and the Myth of the Pharaoh's Curse

Thoth brings the gift of writing, but Thamus sees it as a curseThe Sedona Conference has released its Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery (August 2007) for public comments. A copy may be downloaded for personal use. This Best Practices Commentary, like all of the Sedona publications, was written by a committee of expert members of The Sedona Conference, who agreed upon the content and wording. This particular group is called the "Search and Retrieval Sciences Project Team." Writings by committee are usually an invitation for disaster, but Sedona consistently manages to pull it off, and do a first rate job, primarily, I think, because of the quality of their editors. The Editor-in-Chief for the Search Team is Jason Baron, about whom I have written several times previously, along with Executive Editors Richard Braman and Kenneth Withers, and Senior Editors Thomas Allman, James Daley and George Paul.

The Search Commentary begins by concisely stating the problems faced today to search high volumes of ESI. It then offers three general solutions, followed by eight specific "Practice Points." The comments contain both intellectual depth and good practical advice to all those struggling with the problems of search.

The Search Commentary is carefully considered and well written. Although I have a couple of suggestions on the comments, I fully agree with the committee's observations and solutions. Many will not. In fact, I suspect that this publication will be quite challenging to many in the legal profession because it contradicts several well-established myths. For instance, the Search Team acknowledges that most people consider:

manual review by humans of large amounts of information is as accurate and complete as possible - perhaps even perfect - and constitutes the gold standard by which all searches should be measured.

But the committee states that this is a myth! Manual review may be perfect for a few hundred pages of documents, but fails miserably for a few hundred thousand, much less million, or billion. So much for the gold standard.

The Search Team also make the point, which is not controversial, that the large amounts of ESI in many lawsuits today has made the venerated process of "eyes only review" both impractical and cost-prohibitive. They contend that a new consensus is forming in the legal community:

that human review of documents in discovery is expensive, time consuming, and error-prone. There is growing consensus that the application of linguistic and mathematic-based content analysis, embodied in new forms of search and retrieval technologies, tools, techniques and process in support of the review function can effectively reduce litigation cost, time, and error rates.

This leads to the Practice Point 1 (of 8):

In many settings involving electronically stored information, reliance solely on manual search process for the purpose of finding responsive documents may be infeasible or unwarranted. In such cases, the use of automated search methods should be viewed as reasonable, valuable, and even necessary.

The automated search method of choice today is the almost-as-venerated process of keyword search review. It involves the use of select keywords that you think the documents you are looking for will contain. Keyword searches also frequently include "boolean" logic, and can be expanded further with fuzzy logic, and stemming. You then manually search the documents located by keyword search to determine relevance. The manual review then frequently leads to adjustments in the query terms and repeat of the keyword search. Most lawyers think that with this kind of iterative process, and skilled researchers, you can find most of the documents you are looking for.

In fact, in a study done in 1985, lawyers and paralegals having special skills in this area searched a discovery database of 40,000 documents and 350,000 pages in a case involving a subway accident. David Blair & M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document Retrieval System, 28 Com. A.C.M. 289 (1985). At the end of the lengthy process, the legal team was confident that they had located about 75% of the relevant documents. In my experience, most attorneys think they have a similar, if not better, success rate.

Lawyers have been using keyword searches since the '70s with Lexis and Westlaw to find case law. I was first trained in this in 1978. At that time, Westlaw and Lexis each had mandatory video (VHS) training programs leading to certification. Once certified, you could use "dumb terminals" to access mainframes over modems. It was a tremendous innovation in its day.

It was a natural extension in the '80s and '90s to use the same keyword search technology to locate relevant documents in large sets of ESI. Lawyers and judges quickly endorsed this legal research method to also search for documents. As one judge put it, "the glory of electronic information is not merely that it saves space but that it permits the computer to search for words or 'strings' of text in seconds." In re Lorazepam & Clorazepate, 300 F.Supp.2d 43, 46 (D.D.C. 2004). Keyword searching appeared to solve the problem of large volumes of electronic documents where the gold standard of "eyes only" review was not practical. It might not be perfect like manual searches, but it got at least 75% of the documents, and so was an acceptable alternative.

The profession today is very familiar and comfortable with keyword searching. Keyword search is the method employed by almost all lawyers when they use an automated search process. In fact, I suspect that most lawyers are not even aware that there are alternatives to keyword searches.

That is why the committee's next contention may prove very controversial: the supposed accuracy of keyword searches is just another myth! The Blair and Maron study in 1985 showed that, while the lawyers thought they had found at least 75% of the relevant documents, in fact they had only located 20%.

Can justice really be served with only 20% of the picture? Has the exploding cornucopia of ESI cursed the legal system with the pretence of real knowledge?

The Blair and Moran study, which is still the only one of its kind, led one commentator, Daniel Dabney, a lawyer and information scientist who now works for Westlaw, to equate the false confidence of computer searchers to the Curse of Thamus. Daniel P. Dabney, The Curse of Thamus: An Analysis of Full-Text Legal Document Retrieval, 78 LawLibr. J. 5 (1986). Thamus was an Egyptian Pharaoh reported by Plato in his Phaedrus Dialogue to have criticized the invention of writing as a false substitute for real learning. Thamus condemned writing, said to be a gift from the god Theuth (aka Hermes), as a curse in disguise. The Pharaoh predicted that writing would only lead to a delusionary "semblance of truth" and "conceit of wisdom." As Dabney put it in his article:

Since the mere possession of writings does not give knowledge, how are we to extract from this almost incomprehensibly large collection of written records the knowledge that we need?

Dabney argued that the Blair and Maron study proved that full-text computer assisted retrieval was not a valid cure to the Pharaoh's curse. The Sedona Search Team agrees:

. . . the experience of many litigators is that simple keyword searching alone is inadequate in at least some discovery contexts. This is because simple keyword searches end up being both over- and under-inclusive in light of the inherent malleability and ambiguity of spoken and written English (as well as all other languages). . . .

The problem of the relative percentage of "false positive" hits or noise in the data is potentially huge, amounting in some cases to huge numbers of files which must be searched to find responsive documents. On the other hand, keyword searches have the potential to miss documents that contain a word that has the same meaning as the term used in the query, but is not specified. . . .

Finally, using keywords alone results in a return set of potentially responsive documents that are not weighted and ranked based upon their potential importance or relevance. In other words, each document is considered to have an equal probability of being responsive upon further manual review.

The Sedona Search Team notes that currently most e-discovery vendors and software providers continue to rely on outdated keyword searching. This is also what I am seeing. So, obviously this message may come as an unwelcome challenge to many e-discovery providers, and is therefore likely to be controversial.

But the Sedona Search Commentary does not end on a negative note; instead it points to new search technologies that will significantly improve upon the dismal recall and precision ratios of keyword searches. Here is how they summarize the herald of coming good:

Alternative search tools are available to supplement simple keyword searching and Boolean search techniques. These include using fuzzy logic to capture variations on words; using conceptual searching, which makes use of taxonomies and ontologies assembled by linguists; and using other machine learning and text mining tools that employ mathematical probabilities.

This part of the new Commentary is really interesting, albeit challenging, as the Team talks about alternative search tools and methods, and describes many of them in detail in the Appendix.

The many incredible advances in technology over the last twenty years have created the legal morass we are in now. In our present cursed state, it is impossible to find all relevant evidence, and a mere 20% capture rate seems pretty good. The only viable solution is to fight fire with fire, and find a high-tech answer. This requires a new kind of team synergy that I often talk about in this blog, a combination of Science, Technology and the Law. The Sedona search group concludes with a similar recommendation:

The legal community should support collaborative research with the scientific and academic sectors aimed at establishing the efficacy of a range of automated search and information retrieval methods.

The problems created by the information explosion impact all of society, not just the law. There is strong demand for new, improved search technologies, and this is becoming big business. Billions of dollars are now pouring into search technology research. For instance, in 2006 Google spent $1.23 billion, Yahoo spent $833 million, and e-Bay spent $495 million in core research and development activities. With this kind of commercial activity, there is good reason to hope that the Pharaoh's curse may soon be lifted.

For more information on this subject look for the West Legalworks CLE webinar I did with Jason R. Baron - Director of Litigation, U.S. National Archives and Records Administration, College Park, Maryland; Doug Oard, Ph.D. - Associate Dean for Research, College of Information Studies, University of Maryland; and my law partner at Akerman in Los Angeles, Michael S. Simon. The 1.5 hour audio CLE is entitled The e-Discovery Search Quagmire: New Approaches to the Problem of Finding Relevant Needles in the Electronic Haystack. Also, you may want to see my essay on keyword and concept searches.

Second Edition of The Sedona Principles and the Need for Proportionality

The Sedona Conference this week published the second edition to The Sedona Principles: Best Practices Recommendations & Principles for Addressing Electronic Document Production (June 2007). As a Sedona Conferencemember, I may be somewhat biased, but it is safe to say that everyone in the e-discovery world considers The Sedona Principles a key document for understanding electronic discovery and best practices. This is especially true of the judiciary, who play an active role in Sedona. Many district court judges across the country cite to The Sedona Principles and consider it to be authoritative. The new Principles can be downloaded for free at the Sedona website, so long as the copy is for your personal use only.

If you have not read the first edition published in 2004 (with annotations revised in 2005), I strongly recommend you now go ahead and read this new and improved second edition. To those who have studied the original Sedona Principles, relax, the 14 Principles remain the same, although they have been reworded somewhat. This is a refined and updated version, not a radical rewrite, which is hardly necessary or appropriate after only three years. But, you will still want to download and begin using this new version ASAP. It is a significant improvement over the first edition in several respects.

First, and most importantly, the second edition now ties directly into the new Federal Rules of Civil Procedure, most of which were significantly influenced by the Principles. The interface between the 14 Principles and the new rules is well explained. All of the Principle Commentaries have been updated and refined, but especially the Commentaries for Principle 12 on metadata, and Principle 14 on the imposition of sanctions. The Resources and Authorities provided with each Comment have been updated to include several new e-discovery cases and articles. In the process the total text has grown 30%, from 56 to 73 pages (excluding Appendixes). Finally, the second edition is more user-friendly and better written than before. For instance, it now includes a handy chart in the front that lists Topics correlating to the Principles, Federal Rules, and Sedona Commentary.

The new Preface makes the point that the Conference tried to keep the "rule of reasonableness" foremost in mind when writing and revising these e-discovery principles and best practices.

That rule is embodied in Rule 1 of the Federal Rules of Civil Procedure (courts should secure the just, speedy and inexpensive determination of all matters) and is applied through former Rule 26(b)(2) (now renumbered as Rule 26(b)(2)(C) - proportionality test of burden, cost and need) and in many state counterparts. The rule of reasonableness means that litigants should seek - and the courts should permit - discovery that is reasonable and appropriate to the dispute at hand while not imposing excessive burdens and costs on litigants and the court.

The best practices recommended by Sedona must always be tempered by proportionality. In other words, what may be reasonable for a ten million dollar case may be impractical for a routine case that barely makes the federal jurisdictional minimum. That point is also embodied in Principle 2, which counsels application of the proportionality standard in making a costs-versus-needs analysis to determine what e-discovery efforts are appropriate. The Preface goes on to expand upon this point as follows:

Electronic discovery is a tool to help resolve a dispute and should not be viewed as a strategic weapon to coerce unjust, delayed, or expensive results. The need to act in good faith also extends to the efforts taken to reasonably preserve relevant electronic information, to the form of the production, and to the allocation of the costs of the preservation and production. All discovery issues should be considered in light of the nature of the litigation and the amount in controversy, as well as the cost, burden, and disruption to the parties' operations.

If only all counsel would remember and follow these wise dictates, and judges would enforce them, we would all be better off. The reality is, many litigants are misusing e-discovery as a strategic weapon, including the so-called "weapons of mass discovery" as discussed in my June 20, 2007, Blog on discovery of computer RAM memory.

This is a key point for me, and so I am pleased that the newly revised Comment 2.b. now emphasizes how the total costs of e-discovery must be considered, not just the costs of retrieval and production:

Costs cannot be calculated solely in terms of the expense of computer technicians to retrieve the data but must factor in other litigation costs, including the interruption and disruption of routine business processes and the costs of reviewing the information. Moreover, burdens on information technology personnel and the resources required to review documents for relevance, privilege, confidentiality, and privacy should be considered in any calculus of whether to allow discovery, and, if so, under what terms. In addition, the non-monetary costs (such as the invasion of privacy rights, risks to business and legal confidences, and risks to privileges) should be considered. Evaluating the need to produce electronically stored information often requires that a balance be struck between the burdens and need for electronically stored information, taking into account the technological feasibility and realistic costs involved.

Comment 2.b. at page 17 of Sedona Principles.

This point about hidden costs needed to be emphasized. It has been overlooked multiple times in the last several years, especially by over-reaching plaintiffs and less-discerning judges. The disruption and privacy factors are hard to quantify, but are very real and burdensome. Further, the cost of review is skyrocketing out of control, despite some attempts to export the work to low paid hourly lawyers in India. The attorney fees incurred to review ESI for relevance, privilege, confidentiality, and the like, now constitute the bulk of all e-discovery expenses. The estimates I have seen range from a low of 40% to a high of 60% of total e-discovery costs.

All of these costs, especially review expenses, should, in my opinion, be considered by the courts in any proportionality analysis of whether to allow discovery, or to shift the costs of discovery. These costs should be considered regardless of whether or not the ESI at issue is "reasonably accessible." Unfortunately, Rule 26(b)(2)(B) does not expressly say that. It provides for cost-shifting only if the ESI is inaccessible. For that reason, a request for accessible ESI that would impose unreasonable expenses on the producing party must be opposed on the proportionality test under Rules 1 and 26(b)(2)(C), Federal Rules of Civil Procedure, and Sedona Principle 2. Only in this way can the integrity of the judicial system be protected from the intentional misuse of e-discovery to force settlements based on expense avoidance, instead of merit.


Commentary of Legal Holds

The Sedona Conference Commentary on Legal Holds: The Trigger & The Process (August 2007 Public Comment Version) provides much-needed guidance on an issue very troublesome to most large organizations: when and how to preserve ESI for purposes of litigation. In their words:

The basic principle that an organization has a duty to preserve relevant information in anticipation of litigation is easy to articulate. However, the precise application of that duty can be elusive. Everyday, organizations apply the basic principle to real-world circumstances, confronting the issue of when the obligation is triggered and, once triggered, what is the scope of the obligation. This Article, intended to provide guidance on those issues, is divided into two parts: The "trigger" and the "legal hold."

Once again, this is a group effort by a special committee of The Sedona Conference called the "Working Group on Electronic Document Retention and Production." The Working Group includes many well-known experts on this tricky topic. Under the guidance of the group's editors, Conor Crowley, Eric Schwarz and Gregory Wood, they agreed upon a set of eleven guidelines with detailed commentary on each.

These are common-sense-type principles that will, I predict, be acceptable to most companies, although still challenging for many to implement. It is also likely that these guidelines will often be referred to by courts when reviewing the reasonability of a party's legal hold activities in litigation. If you can show that your organization made a good faith effort to follow these guidelines, then you will be in a strong position to argue that any loss of data that incurred anyway is not sanctionable.

The eleven guidelines include advice and language such as:

Guideline 1 - notice of a "credible threat" of litigation as the trigger of the duty to preserve;

Guideline 2 - the "adoption and consistent implementation" of a written policy as a key factor to show reasonableness and good faith;

Guideline 3 - the need for established procedures to report potential litigation threats;

Guideline 6 - the need to issue a written legal hold notice in most circumstances; and,

Guideline 10 - the need for legal holds, once issued, to thereafter be "regularly monitored."

Guideline 7 provides important input on the scope of a hold, including the key "proportionality" criteria that I have written about before, namely the consideration of the amount in controversy and related factors.

Guideline 7: In determining the scope of information that should be preserved, the nature of the issues raised in the matter, experience in similar circumstances and the amount in controversy are factors that may be considered.

The proportionality factor in determining the proper scope of a hold is often overlooked, especially by some plaintiffs" counsel who still routinely make outrageously overbroad preservation demands.

The comments to Guideline 7 also point out that accessibility is another important factor to consider in determining the proper scope of preservation. In fact, the wording of Guideline 7 in the commentary is slightly different from the wording quoted above that appears at the beginning of the document. They both begin the same, but the version in the comments sections ends with "accessibility of the information" as a factor that may be considered, instead of the first stated "amount in controversy" factor. I think this is just a mistake that will be cleaned up in the final draft.

The comments do, however, make clear that the issue of the costs and burden to try to preserve certain types of data, including, as the comments mention, "voicemail and instant messaging" should be considered. In my view, this means that in many circumstances, not-reasonably-accessible ESI does not need to be preserved. The Sedona comments to Guideline 7 do not, however, go quite that far. They instead make the point, which is not necessarily contrary because it is dependent on the facts, that just because a type of ESI is not-reasonably-accessible, does not mean that it is outside of the scope of a duty to preserve.

We all agree that it depends on the circumstances. In some circumstances, the inaccessibility of ESI will take it outside of the scope of preservation, but in some circumstances it will not. In my view, the general bias should be to exclude not-reasonably-accessible ESI from preservation. As I have mentioned several times previously in prior essays, I think Judge Schiendlin got it right in Zubulake IV when she held:

The scope of a party's preservation obligation can be described as follows: Once a party reasonably anticipates litigation, it must suspend its routine document retention/destruction policy and put in place a litigation hold to ensure the preservation of relevant documents. As a general rule, that litigation hold does not apply to inaccessible backup tapes, for example, typically maintained solely for the purpose of disaster recovery, which may continue to be recycled on the schedule set forth in the company's policy. (emphasis added)

Zubulake v. UBS Warburg LLC, 220 F.R.D. 212, 218 (S.D.N.Y. 2003). Backup tapes are, of course, a prime example of not-reasonably-accessible ESI.

Guideline 8 is the most detailed guideline:

A legal hold is most effective when it:
(a) Identifies the persons who are likely to have relevant information and communicates a preservation notice to those persons;
(b) Communicates the preservation notice in a manner that ensures the recipients will receive actual, comprehensible and effective notice of the requirement to preserve information;
(c) Is in written form;
(d) Clearly defines what information is to be preserved and how the preservation is to be undertaken;
(e) Is periodically reviewed and, when necessary, reissued in either its original or an amended form.

Guideline 8 comes with a sample Certification of Completion document (Appendix B) designed to serve as a checklist for the recipient to confirm that he or she has complied with a hold notice. It requires the recipient to certify to the general counsel's office that they have searched a long list of locations where responsive ESI might be located, including the LAN server for their office, laptop and office computer, email, home computers, Blackberries, email trash bin and desktop recycle bin, "removable storage media, such as disks, CDs, DVDs, memory sticks, and thumb drives," and "files of any administrative personnel working for me."

Many companies will dislike how extensive and complete this list is, especially the inclusion of home computers, removable storage media and secretarial ESI. The commentary softens the blow somewhat by clarifying that this is not a form; it is only a sample, and may not be appropriate for every case. Still, this long list could put a huge search burden on many employees that is not appropriate for many cases, and so this aspect of the commentary is likely to be controversial.

Another controversial aspect of Guideline 8 is its placement of the burden for search and preservation upon the individual employees themselves, instead of on the company, its IT department or automated procedures. This let-the-employees-do-everything procedure has been the norm in the past, but is beginning to be challenged by many courts, especially where there is inadequate follow-up. See Cache La Poudre Feeds, LLC v. Land O'Lakes Farmland Feed, LLC, 2007 WL 684001 (D.Colo. March 2, 2007) and my prior essay on the subject Litigation Hold Is Not Enough: Sanctions Imposed Under Rule 26(g) for Negligent Collection and Preservation


Email Guidelines

The Sedona commentary on email is a short gem edited by Thomas Y. Allman, The Sedona Conference Commentary on Email Management: Guidelines for the Selection of Retention Policy (August 2007). The Commentary begins with a flurry of statistics showing the prevalence of email today, citing to one study estimating that 183 Billion email messages a day were sent in 2006. This is quite a bit larger than the previous estimates I have seen of 60 Billion a day. Whatever, it is huge, and getting bigger every day, and everyone knows the strongest ESI evidence is usually an intemperate email. See my prior essay Top Corporate Officers Continue To Write Embarrassing Emails. That is why most litigation today correctly focuses on email, and why this Email Commentary is so important.

The Sedona working group on email was able to agree on four general guidelines:

Guideline1: Email retention policies should reflect the input of functional and business units through a team approach and should include the entire organization including any operations outside the United States.

As the name of my blog indicates, e-Discovery Team, I obviously think they have it right to emphasize the need for the team approach in forming policies of any kind, not just email. The commentary explains that this should be an interdisciplinary team including Legal, IT and Management, including "Records Management, Compliance, Finance and representatives of major business units." The commentary also mentions that the internal corporate team may want to utilize outside counsel, consultants and vendors. They also recommend that:

A fully engaged responsible person should be appointed to lead the team to work closely on implementation, including recommendations on budget or funding decisions as well as monitoring the program after implementation.

The next Guideline

2 points to the obvious need to understand the actual practices of the organization, as opposed to its written rules on emails, since in most organizations there is a gap between the two that the team needs to address.

Guideline 2: The team should develop a current understanding of email retention polices and practices actually in use within the entity.

The commentary includes a list of useful questions for the team to ask to explore the organization's actual practices.

The third Guideline states:

Guideline 3: An entity should select features for updates or revisions of email retention policy with the understanding that a variety of possible approaches reflecting size, complexity and policy priorities are possible.

The commentaries on Guideline 3 point out that it is usually difficult to reach consensus between the different needs of the groups represented in the team, and suggests open discussion of the differences as the best approach. See Osterman, "E-Mail Archiving Dependent upon Corporate Culture," Network World (March 22, 2005) (describing the resolution of competing views of legal counsel who preferred to purge and a compliance officer who preferred to save email).

To help the team reach a consensus, the Guidelines include an Appendix with two opposite email policy statements. Policy 1 is based on a "short default retention strategy" where emails are only retained for a short period (e.g., 30-90 days) and thereafter are automatically deleted. The user can only avoid this deletion "by taking explicit, affirmative actions" such as moving the email to a dedicated storage area. The other opposite Policy 2 is based on an "indefinite default retention strategy." In this policy, email is retained on active servers for a time, and then "moved automatically to tiered storage and retained indefinitely" or some other specified period such as three to five years.

The fourth Guideline states:

Guideline 4: Any technical solutions should meet the functional requirements identified as part of policy development and should be carefully integrated into existing systems.

Most e-discovery teams studying these issues find that some kind of software and hardware purchases are needed to implement their recommended email practices. This guideline addresses this reality, and counsels the importance of carefully integrating the new purchases into existing systems. The commentary notes that there are many different software and hardware solutions offered by competing vendors to solve these problems, and no one approach is superior. The commentary then provides a list of practical issues to consider when evaluating competing vendor proposals.

My congratulations and thanks to the hard work of "The Sedona Conference Working Group on Electronic Document Retention and Production, eMail Management and Archiving Special Project Team." They have a long name, but their short, concise guidelines and commentary are a very practical and useful work. Every e-discovery team in America struggles with email issues and all will benefit from these pioneering guidelines.



Sedona on Quality

The Sedona Conference Commentary on Achieving Quality in the E-Discovery Process is a must read for anyone seeking to improve their skills in project management, especially in the core functions of search and review. One of its most important insights is that metrics and statistics are now indispensable tools of discovery. The importance of statistics to the law is actually an old insight that has taken a long time to materialize. The Sedona Commentary quotes the great jurist Oliver Wendell Holmes, Jr., who said in 1897:

For the rational study of the law the black letter man may be the man of the present, but the man of the future is the man of statistics and the master of economics.

The future has taken a lot longer to reach most lawyers than Justice Holmes expected. Statistics is still a stranger to most litigators, especially as a tool of discovery. But by the time you finish studying this new Commentary on Quality, you will see that statistics is a powerful tool of the here and now. We can thank the hardworking Sedona Editors-in-Chief for this excellent new publication, Jason R. Baron and Macyl A. Burke. They were assisted by Senior Contributing Editor, Thomas Y. Allman, and Executive Editors, Richard G. Braman and Kenneth J. Withers, with input from Members of Working Group 1. This commentary is a project of The Sedona Conference Working Group on Best Practices for Document Retention and Production. Here is the opening paragraph of the Executive Summary, which should entice you, like it did me, to read more:

The legal profession is at a crossroads: the choice is between continuing to conduct discovery as it has always been practiced in a paper world before the advent of computers, the Internet, and the exponential growth of electronically stored information (ESI) or, alternatively, embracing new ways of thinking in todays digital world. Cost-conscious clients and over-burdened judges are demanding that parties now undertake new approaches to solving litigation problems. The central aim of the present Commentary is to introduce and raise awareness about a variety of processes, tools, techniques, methods, and metrics that fall broadly under the umbrella term quality measures, and that may be of assistance in taming the ESI beast during the various phases of the discovery workflow process. These include greater use of project management, sampling, and other means to verify the accuracy of what constitutes the output of e-discovery. Such collective measures, drawn from a wide variety of scientific and management disciplines, are intended only as an entry-point for further discussion, rather than any type of all-inclusive checklist or cookie-cutter solution to all e-discovery issues.

The truth is, there can be no cookie-cutter solution or all-inclusive checklist for a subject as complex and dynamic as e-discovery. Yet, those of us who specialize in this area get asked for such easy-buttons all of the time. The best that can be hoped for is competency training and the gift of this writing: quality control procedures.

Critique of the Five Reasons Stated for Quality Control

Why is quality so important to the e-discovery process? The obvious answer is to avoid mistakes and the sanctions that can come with mistakes. The Sedona Commentary agrees that such risk management is the primary factor, but then, at page one lists four other reasons quality control is important:

  1. "Failure to employ a quality e-discovery process can result in failure to uncover or disclose relevant evidence which can affect the outcome of litigation." I call this the "adequate recall factor" where you find enough of the truth for justice to be done in a case.
  2. "An inadequate e-discovery process may allow privileged or confidential information to be inadvertently produced." I call this the "adequate confidentiality factor" where you protect enough of the confidential information to meet your client's specifications. Some may be quite paranoid regarding disclosure of any privileged or confidential information, and willing to spend vast sums of money to avoid it. Others may not care as much if a few attorney-client emails, out of thousands, slip through the cracks and are more willing to rely upon the claw-back protections of Evidence Rule 502 to save money.
  3. "Procedures that measure the quality of an e-discovery process allow timely course corrections and provide greater assurance of accuracy, especially of innovative processes." I consider this to be a secondary "adequate recall factor," which allows for quality adjustment of search protocols and other processes as results are measured, new facts uncovered, issues evolve, and insights gained.
  4. "A poorly planned effort can also cost more money in the long run if the deficiencies ultimately require that e-discovery must be redone." I call this the "do-over avoidance factor," where if you do something right the first time, you do not have to pay to do it again. This, for me, is just one aspect of a larger "money-savings factor" that can result from quality processes.

I agree that all five factors are important, but I am inclined to think that the economic savings that can result from quality control are equally important to the e-discovery process as risk management, although perhaps not as obvious. As Justice Holmes said over 100 years ago, we lawyers of his future must not only be men and women of statistics, but also "masters of economics." The savings can not only be realized by avoiding costly do-overs, as the Commentary points out, but also by increasing culling quantity and review speed. Quality controls make it possible to significantly reduce the amount of ESI to review, and to reduce the ESI volume in a manner that is legally defensible. This culling process is key to dramatic reduction of the costs of e-discovery.

The process of intelligent reduction of data size prior to review is an essential component of what is now being called "early case assessment." One of the Senior Editors of the Commentary on Quality, Jason R. Baron, wrote about this type of case assessment culling in another short paper he wrote with Ronni D. Solomon for use at a Sedona Conference Institute CLE entitled Bake Offs, Demos & Kicking the Tires. In spite of this title, it is well worth reading. See especially their Tip 5 where they talk about utilizing keyword "black lists" to reduce ESI size before full review. They also mention the critical need to see the results of keywords, not just hit totals, in order to make an intelligent choice of effective keywords. It is no longer an acceptable practice to choose keywords in the blind. It results in weak culling and thus excessive review. We need to see the results of keyword filtering to be able to aggressively reduce the volume of ESI and still maintain quality. This type of early case assessment culling is critical because the review stage is by far the most expensive step in e-discovery. Anything that cuts the amount of ESI to be reviewed has a direct, substantial impact on the bottom line. Of course, better, faster, and more efficient review of the ESI can also reduce costs. See Bake Offs, Tip 3, and the discussion of clustering tools. With proper quality controls, the costs savings from culling can be realized without sacrificing the other four goals stated in the Commentary on Quality.

In today's economy, the money-saving aspects of quality control are just as important as risk management, and, in my view, more important than the secondary benefits of "adequate recall factors" and "confidentiality factors." E-discovery costs must be significantly reigned in for the civil justice system to avoid the danger of replacement by private arbitration, or worse, by self-help. For that reason, we must leave the old paradigm of total-recall in favor of a more realistic, cost-controlled view. As I often say these days, how much of the truth can a particular dispute afford in view of the constraints of proportionality and Rule 26(b)(2)(C)?

This is the public comment version of the Sedona Commentary on Quality and it is my hope that the final version will place a greater emphasis on the economic savings possible with quality controls in an early case assessment environment. As Justice Holmes said: "It is the province of knowledge to speak and it is the privilege of wisdom to listen."

Four Guiding Principles Behind the Commentary

The Executive Summary also contains an explanation of the four guiding principles behind the commentary:

Principle 1. In cases involving ESI of increasing scope and complexity, the attorney in charge should utilize project management and exercise leadership to ensure that a reasonable process has been followed by his or her legal team to locate responsive material.

Principle 2. Parties should employ reasonable forms or measures of quality at appropriate points in the ediscovery process, consistent with the needs of the case and their legal and ethical responsibilities.

Principle 3. Implementing a well thought out e-discovery process should seek to enhance the overall quality of the production in the form of: (a) reducing the time from request to response; (b) reducing cost; and (c) improving the accuracy and completeness of responses to requests.

Principle 4. Practicing cooperation and striving for greater transparency within the adversary paradigm are key ingredients to obtaining a better quality outcome in e-discovery. Parties should confer early in discovery, including, where appropriate, exchanging information on any quality measures which may be used.

I was pleased to see that cooperation was included as the fourth principle behind quality control. As I explained in my blog, There Can Be No Justice Unless Lawyers Maintain High Ethical Standards, cooperation is an ethical imperative that will necessarily result in substantial costs savings by the avoidance of litigation churning. The Commentary correctly notes that cooperation can also cause a significant increase in quality. All four of these Principles behind Quality are, in turn, based on the fourteen Sedona Principles, but especially on Principle 11, which states:

A responding party may satisfy its good faith obligation to preserve and produce relevant electronically stored information by using electronic tools and processes, such as data sampling, searching, or the use of selection criteria, to identify data reasonably likely to contain relevant information.

For that reason, much of the Commentary on Achieving Quality in the E-Discovery Process is focused on electronic tools and processes, including sampling and other search methods.

The Commentary Is Not A Recipe Book

This Commentary does not purport to give specific advice on how to maintain quality. So, if you are looking for forms and recipes, you will be disappointed. Sedona was quite correct to so limit this project. Here is their explanation, with which I fully agree:

This Commentary is not intended to serve as a comprehensive roadmap covering all possible uses of quality measures and metrics throughout the e-discovery process. The creativity of the vendors and the bar will ensure that concrete applications of quality techniques will be advanced. Nor have we any bias towards particular methods, tools or technologies or a point of view that asserts that sampling or other types of quality measures are invariably required in every type of litigation. Indeed, the drafters believe that the solution to problems created by scale are not solved by technology per se, which is merely a tool, but by better use of team leader skills, project management, and quality measures.

Although the Commentary discusses many tools including my favorite -- sampling -- it does not set out a step-by-step process for handling e-discovery projects. There is too much variability in cases and facts for that, and frankly, it is still too early for any such best practices specifications. Electronic discovery project management is still in its early stages and many of us who have developed methods tend to keep them close to the vest for competitive purposes. Plus, it is not Sedona's role to endorse one particular technique over another, but rather to address issues on a high level and facilitate further study and dialogue. If you read this Commentary with this expectation, you will not be disappointed. Indeed, you will find that it contains many valuable treasures and insights. As Oliver Wendell Holmes, Jr., said: "A moment's insight is sometimes worth a lifetime's experience."


Key Elements of Successful Project Management

The Commentary lists seven elements that Sedona considers critical for successful project management. They do not claim that the list is exhaustive, nor do they purport to rank them in importance. These elements are:

1. Leadership. 2. Tailoring. 3. Expertise. 4. Adaptability. 5. Measurement. 6. Documentation. 7. Transparency.

Although Sedona will not rank these seven, I will. I think that Leadership and Measurement are the two most important factors and the two most difficult. A close third is Documentation, which is key to protection, especially if any of the million decisions made in the course of a project prove to be a mistake. The courts do not expect perfection from attorneys, only expertise, reasonability, and good faith. Good documentation of the process can make it easier to recreate what happened. It will help you to convince the court that you thought you were doing the right thing at the time, even though later events may suggest otherwise. Personally, I do not like the documentation process at all and find it to be as odious as filing. But I know from experience just how valuable it can be, not only to later show what you did, but also in real time to help you to keep track of what you are doing. Leadership is obviously important. Someone has to be in charge of a project and it needs to be an attorney with special expertise. It is a mistake for the trial attorney to try to fill this role because they probably do not have the necessary skills nor time. But it is also a mistake to delegate the job to a non-lawyer. I continue to believe that lawyers must remain masters of the discovery process and they abdicate their responsibilities when they over-delegate to vendors. The Sedona Commentary (at page 7) agrees that attorneys should retain the "Team Leader" position and should only look to outside vendors for competent assistance, which, it correctly states, is "often essential."


The importance of "Measurement" may not be as obvious as Leadership, but the Commentary does a good job of explaining just how indispensible it is to quality control. It lists five types of quality measurements that are especially useful in e-discovery:

1. Judgmental Sampling 2. Independent Testing 3. Reconciliation Techniques 4. Inspection to verify and report discrepancies 5. Statistical Sampling

Of these five measures of quality, the two types of sampling are, in my opinion, the most important. The two types of sampling are defined and explained in detail in Appendix A: Sampling 101 for the E-Discovery Lawyer:

Judgmental Sampling: Sampling performed on a sample set that was selected based on the judgment of the person doing the sampling. ... A common example in the e-discovery context would be keyword searching itself, which is a more-or less informed technique universally used by lawyers and legal professionals to produce a sample slice of a given ESI universe of data, based on the a priori judgment of those selecting the keyword terms.

Statistical Sampling: Probability sampling, or random sampling, is a sampling technique in which the probability of getting any particular sample may be calculated. ... A random sample is one chosen by a method involving an unpredictable component.

In the body of the text at page eleven, the Commentary explains that "statistical sampling can serve as a check on the effectiveness of search terms and other automated tools in identifying responsive information." You can, for instance, use random sampling to test small subsets of the data selected by judgmental sampling, such as keyword culling. Then, by quick reviews of random sample subsets, you can determine the effectiveness of the keywords to identify responsive information. You can then adjust your keywords accordingly and try the new search cull terms again on a new sample. Here is how the Commentary describes this process at page thirteen: "Trial or pilot runs of combinations of words may be tested in an iterative fashion to extrapolate the effectiveness of the chosen set." You can use this iterative method to increase the culling rate of ESI to a size where cost projections of final review expenses finally come within the project budget. This use of both judgmental and random sampling methods, coupled with cost estimations, is close to the kind of early case assessment quality control procedure that I have developed to control e-discovery expenses in a legally defensible manner. I would tell you more, but it gets extremely complicated, is case sensitive, and frankly, leads into proprietary territory.

Applying Quality Measures

Part Three of the Commentary, entitled "Applying Quality Measures in E-Discovery," is probably the section that will be of most interest to practitioners. It divides the analysis into two segments: the Data Collection Phase; and, the Review and Production Phase. The Data Collection Phase is examined in three segments: "Building on Traditional Approaches to Document Collection; Applying Measures of Quality to the Data Collection Process; and, Best Practice Guidelines." The Best Practices Guidelines at page fifteen begins with the following good advice:

The selection, organization and filtering of ESI through the use of a search protocol is a critical element in reducing the volume of information to be collected and thus the time and cost of collection. In addition, keyword search techniques are well known and may be used for this purpose. More advanced technologies have emerged that employ complex algorithms for ESI filtering and organization and may, in some cases, be useful at the collection stage. Regardless of the technology chosen, all filtering methods require a well-defined process. Without these basic steps, the use of any filtering technology will likely result in gross over- or under-inclusion of responsive ESI. The process includes several steps: Ģ Understanding the composition of source ESI; Ģ Defining the goals of the filtering; Ģ Applying the filter and testing the results.

On this last filtration/testing step the Commentary makes the basic, yet important point missed by most practitioners still using negotiated keyword searching, that:

The filtering process should be iterative and needs to be repeated until the desired goals are met. It is not sufficient to blindly run a filtering tool and trust that it is achieving the desired results. One must evaluate the outcome of the search, looking to identify errors in how the filter rules were set up or applied. Key metrics, such as the number of included or excluded documents by keyword or filtering criteria, can be used to evaluate the outcome. Examining the high and low number of search hits can uncover issues with how the search was constructed, the choice of terms, or even issues with the data.

No one is good enough to pick good keywords off the top of their head, much less negotiate a good set of keywords. Words are, after all, so maliable and differ tremendously between person to person. As Oliver Wendell Holmes said:

A word is not a crystal, transparent and unchanging, it is the skin of a living thought and may vary greatly in colour and content according to the circumstances and time in which it is used.

Yet keyword search is still the practice used by most lawyers today and is often ordered by the court. American Family Mutual Ins. Co. v. Gustafson, 2009 WL 641297 at *3 (D.Co. March 10, 2009) ("the parties shall forthwith meet, confer, and agree upon the search terms"). This is a mistake. The "skin of living thoughts" is not so easily snared. Testing of proposed search terms should always be required. Otherwise, your review will be haphazard at best, and a complete waste of time and money at worst. As Oliver Wendall Holmes also said: "Lawyers spend a great deal of their time shoveling smoke." Quality controls in the review and production phase are also examined at length in the Commentary. The discussion includes: automated methods to reduce the initial burden of review; clawback agreements, Rule 502, and reliance on automated methods; quality control guidelines for responsiveness and privilege; and, final quality checking at production.


The Commentary conclusion begins with a quote I like a lot by William Gibson: The future is already here its just not evenly distributed yet. I know many people in this field feel like that is the story of their life. Certainly there is a wide variation in the U.S. and around the world in how the discovery of written evidence is conducted. The Commentary ends with these fine words of wisdom:

In the end, cost-conscious firms, corporations, and institutions of all kinds intent on best practices, as well as over-burdened judges, will demand that parties undertake new ways of thinking about how to solve discovery problems including employing better project management and better measures of quality to achieve optimum results, as outlined here. The technical and management-orientated quality processes discussed above need to be incorporated into every trial lawyers continuing education and daily practice. These processes also dovetail with, and support The Sedona Conference Cooperation Proclamation which calls for incorporation of the best thinking of disciplines outside the law to achieve the goal of the just, speedy, and inexpensive determination of every action. In the end, striving to attain a quality outcome in the conduct of litigation is consistent with the highest ethical calling of the legal profession.

I agree with these noble aspirations, but think it is unrealistic to think that these processes will, or even should, "be incorporated into every trial lawyers continuing education and daily practice." Not every trial lawyer will be interested in random sampling, iteration, linguistic analysis, the latest concept-search engines, ESI architecture, leadership of complex e-discovery projects, and ESI architecture, not to mention the ever changing technologies that create and store electronic information. I agree that all trial lawyers should have some exposure to this and to the idea of quality control, in the same way that all lawyers should have some exposure to antitrust law. But I doubt very much that the subjects in the Quality Commentary are going to be part of "every trial lawyer's daily practice" anytime soon (if ever). Instead, they will be front and center in the practice of attorneys who specialize in e-discovery.

In the inconsistent future here-now that I see, e-discovery specialists will work closely with trial lawyer specialists. Team-work will be common, even on small cases. Some trial lawyers may have the time and inclination to handle e-discovery themselves, especially in less complicated situations. But, for the foreseeable future, they are likely to be few and far between. As William Gibson says: "Time moves in one direction, memory in another." Instead, most trial lawyers will work with, not replace, the e-discovery lawyers. This could be a very small team of just two persons, like Perry Mason and Paul Drake, with a clear division of labor and skills, or, in the largest cases, it could be a team of many lawyers, paralegals, technicians, engineers, and information scientists.

The new age of information is too complicated to continue the old practices and traditions where both trial and discovery skills were combined and held by all trial lawyers. It worked when the documents were paper and few in number. But those days are nearly gone. Now we have ephemeral electronic paper that throws itself away when you are not looking. We have needles of relevant evidence hidden in vast electronic haystacks that are larger and more complicated that you can imagine; haystacks that daily change and grow. As our best experts tell us, search is hard. To use Gibson's words:

I don't have to write about the future. For most people, the present is enough like the future to be pretty scary.

As a consequence, document discovery is far more complicated than it was before and requires special skills to be done right. It is time for the profession to change. As Justice Holmes said:

I find the great thing in this world is not so much where we stand, as in what direction we are moving -- we must sail sometimes with the wind and sometimes against it -- but we must sail, and not drift, nor lie at anchor.

The basics of e-discovery can and should be taught to all trial lawyers. Since they are generally a very smart group, they can learn the basics, if they will take the time and effort needed to do so. (So far, not many have been inclined to make this effort. Most seem to hate e-discovery, but this will change soon.) These basic skills, once learned, can suffice for many small cases, with just an occasional assist from a 21st Century version of Paul Drake. But the larger, more important cases will need the skills of a specialist; skills such as those outlined in The Sedona Conference Commentary on Achieving Quality in the E-Discovery Process.

In today's world of dispute resolution, the client with a sophisticated matter is better served by specialized services with a division of labor. The e-discovery lawyers will possess the skills and quality control techniques discussed in this Commentary, as well as the many other skills discussed in the many other Sedona Commentaries and other books and articles on the subject. These skills take time to learn and time to practice and maintain. There is not enough time to also learn the many, very different skills needed by a trial lawyer. As a result, in complex cases the discovery lawyers will go through the electronic maze to find the facts and their trial lawyer partners will present them to the court and argue their significance.

Litigation is already a team effort in most law firms. This trend will continue to grow and the clients will be better served because of it. Far from being more expensive, as you might think because more people are involved, the discovery and trial lawyer team will save money. The Fannie Mae type cases of outrageous e-discovery expenses only happen when trial lawyers dabble in e-discovery and make huge mistakes. A true specialist will not only do things right the first time, with quality, but do them quicker and less expensively. In this way, the e-discovery teams of the future will help preserve our system of justice by making discovery affordable again.



Below is an excellent video of Sedona Native American Culture and History from You Tube. Sorry, it is not funny, but it is interesting.