Evaluation of Diversification Techniques for Legal Information Retrieval †
2. Legal Document Ranking Using Diversification
2.1. Diversification Overview
- Ranking features, features of legal documents that will be used in the ranking process.
- Distance measures, functions to measure the similarity between two legal documents and the relevance of a query to a given document. We note that in this work, as in information retrieval in general , the term “distance” is used informally to refer to a dissimilarity measure derived from the characteristics describing the objects.
- Diversification heuristics, heuristics to produce a subset of diverse results.
2.2. Ranking Features/Distance Measures
- Document similarity: Various well-known functions from the literature (e.g., Jaccard, cosine similarity, etc.) can be employed for computing the similarity of legal documents. In this work, we choose cosine similarity as a similarity measure; thus, the similarity between documents u and v, with term vectors U and V is:
- Document distance: The distance of two documents is:
- Query document similarity. The relevance of a query q to a given document u can be assigned as the initial ranking score obtained from the IR system, or calculated using the similarity measure, e.g., cosine similarity of the corresponding term vectors:
2.3. Diversification Heuristics
- MMR: Maximal marginal relevance , a greedy method to combine query relevance and information novelty, iteratively constructs the result set S by selecting documents that maximize the following objective function:MMR incrementally computes the standard relevance-ranked list when the parameter and computes a maximal diversity ranking among the documents in N when . For intermediate values of , a linear combination of both criteria is optimized. In MMR Algorithm 1, the set S is initialized with the document that has the highest relevance to the query. Since the selection of the first element has a high impact on the quality of the result, MMR often fails to achieve optimum results.
Algorithm 1 Produce diverse set of results with MMR. Input: Set of candidate results N , size of diverse set k Output: Set of diverse results ▹ Initialize with the highest relevant to the query document Set Set while do Find ▹ Iteratively select document that maximize Equation (5) Set Set end while
- Max-sum: The Max-sum diversification objective function  aims at maximizing the sum of the relevance and diversity in the final result set. This is achieved by a greedy approximation, Algorithm 2, that selects a pair of documents that maximizes Equation (6) in each iteration.Max-sum Algorithm 2, at each step, examines the pairwise distances of the candidate items N and selects the pair with the maximum pairwise distance, to insert into the set of diverse items S.
Algorithm 2 Produce diverse set of results with Max-sum. Input: Set of candidate results N, size of diverse set k Output: Set of diverse results for do Find ▹ Select pair of docs that maximize Equation (6) Set Set end for if k is odd then ▹ If k is odd add an arbitrary document to S end if
- Max-min: The Max-min diversification objective function  aims at maximizing the minimum relevance and dissimilarity of the selected set. This is achieved by a greedy approximation, Algorithm 3, that initially selects a pair of documents that maximize Equation (7) and then in each iteration selects the document that maximizes Equation (8):Max-min Algorithm 3, at each step, it finds, for each candidate document, its closest document belonging to S and calculates their pairwise distance . The candidate document that has the maximum distance is inserted into S.
Algorithm 3 Produce diverse set of results with Max-min. Input: Set of candidate results N, size of diverse set k Output: Set of diverse results Find ▹ Initially selects documents that maximize Equation (7) Set while do Find ▹ Select document that maximize Equation (8) Set end while
- Mono-objective: Mono-objective  combines the relevance and the similarity values into a single value for each document. It is defined as:Algorithm 4 approximates the Mono-objective. The algorithm, at the initialization step, calculates a distance score for each candidate document. The objective function weights each document’s similarity to the query with the average distance of the document with the rest of the documents. After the initialization step, where scores are calculated, they are not updated after each iteration of the algorithm. Therefore, each step consists of selecting the document from the remaining candidates set with the maximum score and inserting it into S.
Algorithm 4 Produce diverse set of results with Mono-objective. Input: Set of candidate results N, size of diverse set k Output: Set of diverse results for do ▹ Calculate scores based on Equation (9) end for while do Find ▹ Sort and select documents Set Set end while
- LexRank: LexRank  is a stochastic graph-based method for computing the relative importance of textual units. A document is represented as a network of inter-related sentences, and a connectivity matrix based on intra-sentence similarity is used as the adjacency matrix of the graph representation of sentences.In our setting, instead of sentences, we use documents that are in the initial retrieval set N for a given query. In this way, instead of building a graph using the similarity relationships among the sentences based on an input document, we utilize document similarity on the result set. If we consider documents as nodes, the result set document collection can be modeled as a graph by generating links between documents based on their similarity score as in Equation (2). Typically, low values in this matrix can be eliminated by defining a threshold so that only significantly similar documents are connected to each other. However, as in all discretization operations, this means information loss. Instead, we choose to utilize the strength of the similarity links. This way we use the cosine values directly to construct the similarity graph, obtaining a much denser, but weighted graph. Furthermore, we normalize our adjacency matrix B, so as to make the sum of each row equal to one.Thus, in LexRank scoring formula Equation (10), Matrix B captures pairwise similarities of the documents, and square matrix A, which represents the probability of jumping to a random node in the graph, has all elements set to , where is the number of documents.LexRank Algorithm 5 applies a variation of PageRank  over a document graph. A random walker on this Markov chain chooses one of the adjacent states of the current state with probability or jumps to any state in the graph, including the current state, with probability λ.
Algorithm 5 Produce diverse set of results with LexRank. Input: Set of candidate results N, size of diverse set k Output: Set of diverse results for do ▹ Calculate connectivity matrix based on document similarity Equation (2) end for ▹ Calculate stationary distribution of Equation (10). (Omitted for clarity) while do Find ▹ Sort and select documents Set Set end while
- Biased LexRank: Biased LexRank  provides for a LexRank extension that takes into account a prior document probability distribution, e.g., the relevance of documents to a given query. The Biased LexRank scoring formula is analogous to LexRank scoring formula Equation (10), with matrix A, which represents the probability of jumping to a random node in the graph, proportional to the query document relevance.Algorithm 5 is also used to produce a diversity-oriented ranking of results with the Biased LexRank method. In Biased LexRank scoring formula Equation (10), we set matrix B as the connectivity matrix based on document similarity for all documents that are in the initial retrieval set N for a given query and matrix A elements proportional to the query document relevance.
- DivRank: DivRank  balances popularity and diversity in ranking, based on a time-variant random walk. In contrast to PageRank , which is based on stationary probabilities, DivRank assumes that transition probabilities change over time; they are reinforced by the number of previous visits to the target vertex. If is the transition probability from any vertex u to vertex v at time T, is the prior distribution that determines the preference of visiting vertex and is the transition probability from u to v prior to any reinforcement, then,DivRank was originally proposed in a query independent context; thus, it is not directly applicable to the diversification of search results. We introduce a query dependent prior and thus utilize DivRank as a query-dependent ranking schema. In our setting, we use documents that are in the initial retrieval set N for a given query q, create the citation network between those documents and apply the DivRank Algorithm 6 to select divers’ documents in S.
Algorithm 6 Produce diverse set of results with DivRank. Input: Set of candidate results N, size of diverse set k Output: Set of diverse results for do ▹ Connectivity matrix is based on citation network adjacency matrix end for ▹ Calculate stationary distribution of Equation (11). (Omitted for clarity) while do Find ▹ Sort and select documents Set Set end while
- Grasshopper: Similar to the DivRank ranking algorithm, it is described in . This model starts with a regular time-homogeneous random walk, and in each step, the vertex with the highest weight is set as an absorbing state.Since Grasshopper and DivRank utilize a similar approach and will ultimately present rather similar results, we utilized Grasshopper distinctively from DivRank. In particularly, instead of creating the citation network of documents belonging to the initial result set, we form the adjacency matrix based on document similarity, as previously explained in LexRank Algorithm 5.
3. Experimental Setup
3.1. Legal Corpus
3.2. Evaluation Metrics
- a-nDCG: The a-normalized discounted cumulative gain  metric quantifies the amount of unique aspects of the query q that are covered by the ranked documents. We use , as typical in TREC evaluation.
- ERR-IA: Expected reciprocal rank-intent aware  is based on inter-dependent ranking. The contribution of each document is based on the relevance of documents ranked above it. The discount function is therefore not just dependent on the rank, but also on the relevance of previously ranked documents.
- S-recall: Subtopic-recall  is the number of unique aspects covered by the results, divided by the total number of aspects. It measures the aspect coverage for a given result list at depth k.
3.3. Relevance Judgments
4. Related Work
4.1. Query Result Diversification
4.2. Diversified Ranking on Graphs
4.3. Legal Text Retrieval
Conflicts of Interest
- Alces, K.A. Legal diversification. Columbia Law Rev. 2013, 113, 1977–2038. [Google Scholar]
- Koniaris, M.; Anagnostopoulos, I.; Vassiliou, Y. Diversifying the Legal Order. In Proceedings of the IFIP International Conference on Artificial Intelligence Applications and Innovations, Thessaloniki, Greece, 16–18 September 2016; pp. 499–509.
- Carbonell, J.; Goldstein, J. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, Melbourne, Australia, 24–28 August 1988; pp. 335–336.
- Gollapudi, S.; Sharma, A. An Axiomatic Approach for Result Diversification. In Proceedings of the 18th International Conference on World Wide Web, Madrid, Spain, 20–24 April 2009; pp. 381–390.
- Erkan, G.; Radev, D.R. LexRank: Graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res. 2004, 22, 457–479. [Google Scholar]
- Otterbacher, J.; Erkan, G.; Radev, D.R. Biased LexRank: Passage retrieval using random walks with question-based priors. Inf. Process. Manag. 2009, 45, 42–54. [Google Scholar] [CrossRef]
- Mei, Q.; Guo, J.; Radev, D. Divrank: The interplay of prestige and diversity in information networks. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, 25–28 July 2010; pp. 1009–1018.
- Zhu, X.; Goldberg, A.B.; Van Gael, J.; Andrzejewski, D. Improving Diversity in Ranking using Absorbing Random Walks. In Proceedings of the Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2007, Rochester, NY, USA, 22–27 April 2007; pp. 97–104.
- Hand, D.J.; Mannila, H.; Smyth, P. Principles of Data Mining; MIT Press: Cambridge, MA, USA, 2001. [Google Scholar]
- Wong, S.M.; Raghavan, V.V. Vector space model of information retrieval: A reevaluation. In Proceedings of the 7th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Cambridge, UK, 2–6 July 1984; pp. 167–185.
- Page, L.; Brin, S.; Motwani, R.; Winograd, T. The PageRank Citation Ranking: Bringing Order to the Web; Technical Report for Stanford InfoLab: Stanford University, Stanford, CA, USA, 1999. [Google Scholar]
- Galgani, F.; Compton, P.; Hoffmann, A. Combining different summarization techniques for legal text. In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, Avignon, France, 22 April 2012; pp. 115–123.
- Sanderson, M. Test Collection Based Evaluation of Information Retrieval Systems. Found. Trends® Inf. Retr. 2010, 4, 247–375. [Google Scholar] [CrossRef]
- Radlinski, F.; Bennett, P.N.; Carterette, B.; Joachims, T. Redundancy, diversity and interdependent document relevance. In ACM SIGIR Forum; Association for Computing Machinery (ACM): New York, NY, USA, 2009; Volume 43, pp. 46–52. [Google Scholar]
- Clarke, C.L.A.; Kolla, M.; Cormack, G.V.; Vechtomova, O.; Ashkan, A.; Büttcher, S.; MacKinnon, I. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, 20–24 July 2008; pp. 659–666.
- Chapelle, O.; Metlzer, D.; Zhang, Y.; Grinspan, P. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and Knowledge Management—CIKM ’09, Hong Kong, China, 2–6 November 2009; pp. 621–630.
- Zhai, C.X.; Cohen, W.W.; Lafferty, J. Beyond independent relevance. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, Toronto, ON, Canada, 28 July–1 August 2003; pp. 10–17.
- Blei, D.M.; Ng, A.Y.; Jordan, M.I. Latent dirichlet allocation. J. Mach. Learn. Res. 2003, 3, 993–1022. [Google Scholar]
- Singh, J.; Nejdl, W.; Anand, A. History by Diversity: Helping Historians Search News Archives. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, Carrboro, NC, USA, 13–17 March 2016; pp. 183–192.
- Giannopoulos, G.; Koniaris, M.; Weber, I.; Jaimes, A.; Sellis, T. Algorithms and criteria for diversification of news article comments. J. Intell. Inf. Syst. 2015, 44, 1–47. [Google Scholar] [CrossRef]
- Cheng, S.; Arvanitis, A.; Chrobak, M.; Hristidis, V. Multi-Query Diversification in Microblogging Posts. In Proceedings of the 17th International Conference on Extending Database Technology (EDBT), Athens, Greece, 24–28 March 2014; pp. 133–144.
- Koniaris, M.; Giannopoulos, G.; Sellis, T.; Vasileiou, Y. Diversifying microblog posts. In Proceedings of the International Conference on Web Information Systems Engineering, Thessaloniki, Greece, 12–14 October 2014; pp. 189–198.
- Song, K.; Tian, Y.; Gao, W.; Huang, T. Diversifying the image retrieval results. In Proceedings of the 14th ACM International Conference on Multimedia, Santa Barbara, CA, USA, 23–27 October 2006; pp. 707–710.
- Ziegler, C.N.; McNee, S.M.; Konstan, J.A.; Lausen, G. Improving recommendation lists through topic diversification. In Proceedings of the 14th International Conference on World Wide Web, Chiba, Japan, 10–14 May 2005; pp. 22–32.
- Raman, K.; Shivaswamy, P.; Joachims, T. Online learning to diversify from implicit feedback. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China, 12–16 August 2012; pp. 705–713.
- Makris, C.; Plegas, Y.; Stamatiou, Y.C.; Stavropoulos, E.C.; Tsakalidis, A.K. Reducing Redundant Information in Search Results Employing Approximation Algorithms. In Proceedings of the International Conference on Database and Expert Systems Applications, Munich, Germany, 1–4 September 2014; pp. 240–247.
- Zhang, B.; Li, H.; Liu, Y.; Ji, L.; Xi, W.; Fan, W.; Chen, Z.; Ma, W.Y. Improving web search results using affinity graph. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil, 15–19 August 2005; pp. 504–511.
- Chen, H.; Karger, D.R. Less is more: probabilistic models for retrieving fewer relevant documents. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA, 6–10 August 2006; pp. 429–436.
- Cronen-Townsend, S.; Croft, W.B. Quantifying Query Ambiguity. In Proceedings of the Second International Conference on Human Language Technology Research, San Diego, CA, USA, 24–27 March 2002; pp. 104–109.
- Santos, R.L.T.; Macdonald, C.; Ounis, I. Search Result Diversification. Found. Trends® Inf. Retr. 2015, 9, 1–90. [Google Scholar] [CrossRef]
- Drosou, M.; Pitoura, E. Search result diversification. ACM SIGMOD Rec. 2010, 39, 41. [Google Scholar] [CrossRef]
- Santos, R.L.; Macdonald, C.; Ounis, I. Exploiting query reformulations for web search result diversification. In Proceedings of the 19th International Conference on World Wide Web, Hong Kong, China, 1–5 May 2010; pp. 881–890.
- Agrawal, R.; Gollapudi, S.; Halverson, A.; Ieong, S. Diversifying search results. In Proceedings of the second ACM international conference on web search and data mining, Barcelona, Spain, 9–11 February 2009; pp. 5–14.
- Hu, S.; Dou, Z.; Wang, X.; Sakai, T.; Wen, J.R. Search Result Diversification Based on Hierarchical Intents. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Melbourne, Australia, 19–23 October 2015; pp. 63–72.
- Langville, A.N.; Meyer, C.D. A survey of eigenvector methods for web information retrieval. SIAM Rev. 2005, 47, 135–161. [Google Scholar] [CrossRef]
- Moens, M. Innovative techniques for legal text retrieval. Artif. Intell. Law 2001, 9, 29–57. [Google Scholar] [CrossRef]
- Biagioli, C.; Francesconi, E.; Passerini, A.; Montemagni, S.; Soria, C. Automatic semantics extraction in law documents. In Proceedings of the 10th international conference on Artificial intelligence and law, Bologna, Italy, 6–11 June 2005.
- Mencia, E.L.; Fürnkranz, J. Efficient pairwise multilabel classification for large-scale problems in the legal domain. In Machine Learning and Knowledge Discovery in Databases; Springer: Berlin, Germany, 2008; pp. 50–65. [Google Scholar]
- Grabmair, M.; Ashley, K.D.; Chen, R.; Sureshkumar, P.; Wang, C.; Nyberg, E.; Walker, V.R. Introducing LUIMA: an experiment in legal conceptual retrieval of vaccine injury decisions using a UIMA type system and tools. In Proceedings of the 15th International Conference on Artificial Intelligence and Law, San Diego, CA, USA, 8–12 June 2015; pp. 69–78.
- Saravanan, M.; Ravindran, B.; Raman, S. Improving legal information retrieval using an ontological framework. Artif. Intelli. Law 2009, 17, 101–124. [Google Scholar] [CrossRef]
- Schweighofer, E.; Liebwald, D. Advanced lexical ontologies and hybrid knowledge based systems: First steps to a dynamic legal electronic commentary. Artif. Intell. Law 2007, 15, 103–115. [Google Scholar] [CrossRef]
- Sagri, M.T.; Tiscornia, D. Metadata for content description in legal information. In Proceedings of the 14th International Workshop on Database and Expert Systems Applications, Prague, Czech Republic, 1–5 September 2003; pp. 745–749.
- Klein, M.C.; Van Steenbergen, W.; Uijttenbroek, E.M.; Lodder, A.R.; van Harmelen, F. Thesaurus-based Retrieval of Case Law. In Proceedings of the 2006 conference on Legal Knowledge and Information Systems: JURIX 2006: The Nineteenth Annual Conference, Paris, France, 8 December 2006; Volume 152, p. 61.
- Hoekstra, R.; Breuker, J.; di Bello, M.; Boer, A. The LKIF Core ontology of basic legal concepts. In Proceedings of the 2nd Workshop on Legal Ontologies and Artificial Intelligence Techniques (LOAIT 2007), Stanford, CA, USA, 4 June 2007.
- Farzindar, A.; Lapalme, G. Legal text summarization by exploration of the thematic structures and argumentative roles. In Text Summarization Branches Out Workshop Held in Conjunction with ACL; Association for Computational Linguistics (ACL): Barcelona, Spain, 25–26 July 2004; pp. 27–34. [Google Scholar]
- Farzindar, A.; Lapalme, G. Letsum, an automatic legal text summarizing system. In Proceedings of the Legal Knowledge and Information Systems. JURIX 2004: The Seventeenth Annual Conference, Berlin, Germany, 8–10 December 2004; pp. 11–18.
- Moens, M.F. Summarizing court decisions. Inf. Process. Manag. 2007, 43, 1748–1764. [Google Scholar] [CrossRef]
- Aktolga, E.; Ros, I.; Assogba, Y. Detecting outlier sections in us congressional legislation. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval—SIGIR ’11, Beijing, China, 24–28 July 2011; pp. 235–244.
- Marx, S.M. Citation networks in the law. Jurimetr. J. 1970, 10, 121–137. [Google Scholar]
- Van Opijnen, M. Citation Analysis and Beyond: In Search of Indicators Measuring Case Law Importance. In Proceedings of the Legal Knowledge and Information Systems-JURIX 2012: The Twenty-Fifth Annual Conference, Amsterdam, The Netherlands, 17–19 December 2012; pp. 95–104.
- Fowler, J.H.; Jeon, S. The Authority of Supreme Court precedent. Soc. Netw. 2008, 30, 16–30. [Google Scholar] [CrossRef]
- Fowler, J.H.; Johnson, T.R.; Spriggs, J.F.; Jeon, S.; Wahlbeck, P.J. Network Analysis and the Law: Measuring the Legal Importance of Precedents at the U.S. Supreme Court. Political Anal. 2006, 15, 324–346. [Google Scholar] [CrossRef]
- Galgani, F.; Compton, P.; Hoffmann, A. Citation based summarisation of legal texts. In Proceedings of the PRICAI 2012: Trends in Artificial Intelligence, Kuching, Malaysia, 3–7 September 2012; pp. 40–52.
- Koniaris, M.; Anagnostopoulos, I.; Vassiliou, Y. Network Analysis in the Legal Domain: A complex model for European Union legal sources. arXiv, 2015; arXiv:1501.05237. [Google Scholar]
- Lettieri, N.; Altamura, A.; Faggiano, A.; Malandrino, D. A computational approach for the experimental study of EU case law: analysis and implementation. Soc. Netw. Anal. Min. 2016, 6. [Google Scholar] [CrossRef]
- Koniaris, M.; Anagnostopoulos, I.; Vassiliou, Y. Multi-dimension Diversification in Legal Information Retrieval. In Proceedings of the International Conference on Web Information Systems Engineering, Shanghai, China, 8–10 November 2016; pp. 174–189.
- Wittfoth, A.; Chung, P.; Greenleaf, G.; Mowbray, A. AustLII’s Point-in-Time Legislation System: A generic PiT system for presenting legislation. Available online: http://portsea.austlii.edu.au/pit/papers/PiT_background_2005.rtf (accessed on 25 January 2017).
|algorithms tested||MMR, Max-min, Max-sum, Mono-objective, LexRank, Biased LexRank, DivRank, Grasshopper|
|tradeoff l values||0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9|
|candidate set size n =||100|
|result set size k =||5, 10, 20, 30|
|# of sample queries||298|
|1:||Abandoned and Lost Property||3:||Abortion and Birth Control|
|24:||Aliens Immigration and Citizenship||31:||Antitrust and Trade Regulation|
|61:||Breach of Marriage Promise||84:||Commodity Futures Trading Regulation|
|88:||Compromise and Settlement||199:||Implied and Constructive Contracts|
|291:||Privileged Communications and Confidentiality||363:||Threats Stalking and Harassment|
|1||court applicant property respondent claim order costs company trustee trust|
|2||evidence agreement contract business drtime hamiltonrespondent applicant sales|
|3||tribunal rights land title native evidence area claim interests appellant|
|4||evidence agreement meeting conduct time tiltformsecond brand septemberaustralia|
|5||evidence professor university dr property gray patent uwaclaimnrc|
|Document||Topic 1||Topic 2||Topic 3||Topic 4||Topic 5|
|08_711||0.731||0.267||0.00172||8.79 × 10||4 × 10|
|09_1395||0.467||2.25 × 10||0.459||0.0746||1.04 × 10|
|09_1383||0.99||2.85 × 10||0.00944||0.000298||1.31 × 10|
|06_1169||0.994||5.33 × 10||0.00559||5.41 × 10||2.46 × 10|
|07_924||0.237||4.83 × 10||4.64 × 10||4.9 × 10||0.763|
|MMR||0.5187||0.5785 *||0.642 *||0.6676 *||0.5041||0.5341||0.5559 °||0.562 °||0.6145||0.7875 *||0.9135 *||0.9543 *|
|Baseline ()||MMR: Light Diversity ()||MMR: Moderate Diversity ()||MMR: High Diversity ()|
|Query 24: Aliens Immigration and Citizenship|
|1||Virgin Holdings SA v Commissioner of Taxation  FCA 1503 (10 October 2008)||Virgin Holdings SA v Commissioner of Taxation  FCA 1503 (10 October 2008)||Virgin Holdings SA v Commissioner of Taxation  FCA 1503 (10 October 2008)||Virgin Holdings SA v Commissioner of Taxation  FCA 1503 (10 October 2008)|
|2||Undershaft (No 1) Limited v Commissioner of Taxation  FCA 41 (3 February 2009)||Fowler v Commissioner of Taxation  FCA 528 (21 April 2008)||Fowler v Commissioner of Taxation  FCA 528 (21 April 2008)||Soh v Commonwealth of Australia  FCA 520 (18 April 2008)|
|3||Fowler v Commissioner of Taxation  FCA 528 (21 April 2008)||Wight v Honourable Chris Pearce, MP, Parliamentary Secretary to the Treasurer  FCA 26 (29 January 2007)||Coleman v Minister for Immigration & Citizenship  FCA 1500 (27 September 2007)||SZJDI v Minister for Immigration & Citizenship (No. 2)  FCA 813 (16 May 2008)|
|4||Wight v Honourable Chris Pearce, MP, Parliamentary Secretary to the Treasurer  FCA 26 (29 January 2007)||Undershaft (No 1) Limited v Commissioner of Taxation  FCA 41 (3 February 2009)||Charlie v Minister for Immigration and Citizenship  FCA 1025 (10 July 2008)||Charlie v Minister for Immigration and Citizenship  FCA 1025 (10 July 2008)|
|5||Coleman v Minister for Immigration & Citizenship  FCA 1500 (27 September 2007)||Coleman v Minister for Immigration & Citizenship  FCA 1500 (27 September 2007)||VSAB v Minister for Immigration and Multicultural and Indigenous Affairs  FCA 239 (17 March 2006)||VSAB v Minister for Immigration and Multicultural and Indigenous Affairs  FCA 239 (17 March 2006)|
|Query 84: Commodity Futures Trading Regulation|
|1||BHP Billiton Iron Ore Pty Ltd. v The National Competition Council  FCA 1764 (18 December 2006)||BHP Billiton Iron Ore Pty Ltd. v The National Competition Council  FCA 1764 (18 December 2006)||BHP Billiton Iron Ore Pty Ltd. v The National Competition Council  FCA 1764 (18 December 2006)||BHP Billiton Iron Ore Pty Ltd. v The National Competition Council  FCA 1764 (18 December 2006)|
|2||Australian Securities & Investments Commission v Lee  FCA 918 (15 June 2007)||Australian Securities & Investments Commission v Lee  FCA 918 (15 June 2007)||Australian Securities & Investments Commission v Lee  FCA 918 (15 June 2007)||Australian Competition and Consumer Commission v Dally M Publishing and Research Pty Limited  FCA 1220 (10 August 2007)|
|3||Woodside Energy Ltd. (ABN 63 005 482 986) v Commissioner of Taxation (No 2)  FCA 1961 (10 December 2007)||Woodside Energy Ltd. (ABN 63 005 482 986) v Commissioner of Taxation (No 2)  FCA 1961 (10 December 2007)||Woodside Energy Ltd. (ABN 63 005 482 986) v Commissioner of Taxation (No 2)  FCA 1961 (10 December 2007)||Heritage Clothing Pty Ltd. trading as Peter Jackson Australia v Mens Suit Warehouse Direct Pty Ltd. trading as Walter Withers  FCA 1775 (28 November 2008)|
|4||BHP Billiton Iron Ore Pty Ltd. v National Competition Council (No 2)  FCA 557 (19 April 2007)||Keynes v Rural Directions Pty Ltd. (No 2) (includes Corrigendum dated 16 July 2009)  FCA 567 (3 June 2009)||Keynes v Rural Directions Pty Ltd. (No 2) (includes Corrigendum dated 16 July 2009)  FCA 567 (3 June 2009)||Travelex Limited v Commissioner of Taxation (Corrigendum dated 4 February 2009)  FCA 1961 (19 December 2008)|
|5||Keynes v Rural Directions Pty Ltd. (No 2) (includes Corrigendum dated 16 July 2009)  FCA 567 (3 June 2009)||Queanbeyan City Council v ACTEW Corporation Limited  FCA 943 (24 August 2009)||Heritage Clothing Pty Ltd. trading as Peter Jackson Australia v Mens Suit Warehouse Direct Pty Ltd trading as Walter Withers  FCA 1775 (28 November 2008)||Ashwick (Qld) No 127 Pty Ltd. (ACN 010 577 456) v Commissioner of Taxation  FCA 1388 (26 November 2009)|
|Query 291:Privileged Communications and Confidentiality|
|1||Siam Polyethylene Co Ltd. v Minister of State for Home Affairs (No 3)  FCA 839 (7 August 2009)||Siam Polyethylene Co Ltd. v Minister of State for Home Affairs (No 3)  FCA 839 (7 August 2009)||Siam Polyethylene Co Ltd. v Minister of State for Home Affairs (No 3)  FCA 839 (7 August 2009)||Siam Polyethylene Co Ltd. v Minister of State for Home Affairs (No 3)  FCA 839 (7 August 2009)|
|2||AWB Limited v Australian Securities and Investments Commission  FCA 1877 (11 December 2008)||AWB Limited v Australian Securities and Investments Commission  FCA 1877 (11 December 2008)||AWB Limited v Australian Securities and Investments Commission  FCA 1877 (11 December 2008)||Krueger Transport Equipment Pty Ltd. v Glen Cameron Storage  FCA 803 (30 May 2008)|
|3||Brookfield Multiplex Limited v International Litigation Funding Partners Pte Ltd. (No 2)  FCA 449 (6 May 2009)||Brookfield Multiplex Limited v International Litigation Funding Partners Pte Ltd (No 2)  FCA 449 (6 May 2009)||Autodata Limited v Boyce’s Automotive Data Pty Limited  FCA 1517 (4 October 2007)||Futuretronics.com.au Pty Limited v Graphix Labels Pty Ltd.  FCA 1621 (29 October 2007)|
|4||Cadbury Schweppes Pty Ltd. (ACN 004 551 473) v Amcor Limited (ACN 000 017 372)  FCA 88 (19 February 2008)||Barrett Property Group Pty Ltd. v Carlisle Homes Pty Ltd (No 2)  FCA 930 (17 June 2008)||Barrett Property Group Pty Ltd. v Carlisle Homes Pty Ltd (No 2)  FCA 930 (17 June 2008)||Australian Competition & Consumer Commission v Visy Industries  FCA 136 (23 February 2006)|
|5||Barrett Property Group Pty Ltd v Carlisle Homes Pty Ltd. (No 2)  FCA 930 (17 June 2008)||Cadbury Schweppes Pty Ltd. (ACN 004 551 473) v Amcor Limited (ACN 000 017 372)  FCA 88 (19 February 2008)||Optus Networks Ltd. v Telstra Corporation Ltd (No. 2) (includes Corrigendum dated 7 July 2009)  FCA 422 (9 July 2009)||IO Group Inc v Prestige Club Australasia Pty Ltd. (No 2)  FCA 1237 (11 August 2008)|
© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ( http://creativecommons.org/licenses/by/4.0/).
Koniaris, M.; Anagnostopoulos, I.; Vassiliou, Y. Evaluation of Diversification Techniques for Legal Information Retrieval. Algorithms 2017, 10, 22. https://doi.org/10.3390/a10010022
Koniaris M, Anagnostopoulos I, Vassiliou Y. Evaluation of Diversification Techniques for Legal Information Retrieval. Algorithms. 2017; 10(1):22. https://doi.org/10.3390/a10010022Chicago/Turabian Style
Koniaris, Marios, Ioannis Anagnostopoulos, and Yannis Vassiliou. 2017. "Evaluation of Diversification Techniques for Legal Information Retrieval" Algorithms 10, no. 1: 22. https://doi.org/10.3390/a10010022