Welcome to the Data Analytics Platform for "Indian Contracts in Adjudicated Texts ('ICAT')"

What is ICAT? It is a dynamically generated list of Indian judgments and contractual clauses that appear in them via user queries to the app. This is meant to act as a repository of precedents for lawyers who want to know whether they should anticipate any legal issues with specific kinds of contracts. However, this is not the only use-case and there are further integrations using similarity search and sentiment-classification of the clauses being explored. Where can I access the app for making queries? Go to the following link - https://huggingface.co/spaces/schematise/ICAT-query-database-beta Please note that while there is no personal data collected, the database anonymously stores your search queries as well as the results for future training. How is it different from searching on a general purpose database? In the current version of ICAT, you are offered a list of results relevant to your search query, while discarding the others. This list of results is generated by going through an algorithm which also involves a machine learning model for text-classification that sorts the results as required.

The following is a list of search queries entered so far

DROP VIEW IF EXISTS search_query_view; CREATE VIEW search_query_view AS SELECT DISTINCT searchquery FROM search_queries; SELECT * FROM search_query_view

The following table randomly lists 10 results from the database

SELECT title, searchquery, CONCAT(matching_indents_after_classification, matching_columns_after_classification) AS matching_clauses FROM classified_index WHERE searchquery IN (SELECT searchquery FROM search_query_view) AND LENGTH(CONCAT(matching_indents_after_classification, matching_columns_after_classification)) > 10 ORDER BY RANDOM() LIMIT 10;

The following table and visualisation show the year of the decision for a random selection of 100 judgments from all results

SELECT title, SUBSTRING(title FROM '.* ([0-9]{4})$') AS year FROM stored_results WHERE doc_id IN (SELECT doc_id FROM classified_index) ORDER BY RANDOM() LIMIT 100;

This chart is empty

Chart was probably not set up properly in the notebook

Experimental: Conducting Topic Modelling on a subset of contract clauses using BERTopic

Topic Modeling allows one to see what are the most frequently occurring words in a dataset and arrange them by topics or by documents. Hence, BERTopic's integration with vector embeddings is one that is utilised here (the embeddings were generated using an open source domain specific model provided by LLMWare.

import pandas as pd import ast import numpy as np df=pd.read_csv("/work/democsv_table_bert_numpy.csv") df.replace("{}", np.nan, inplace=True) df.dropna(subset=['matching_indents_after_classification'], inplace=True) df

docs=df['matching_indents_after_classification'].tolist() embeddings = df['embedding'].tolist() embeddings_list = embeddings for i in range(len(embeddings)): embeddings_list[i] = eval(embeddings[i]) embeddings_array = np.array(embeddings, dtype="float")

from bertopic import BERTopic from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction import text topic_model = BERTopic() topics, probs = topic_model.fit_transform(docs, embeddings_array) stop_words = text.ENGLISH_STOP_WORDS.union(["agreement", "shall"]) list_of_stop_words = list(stop_words) vectorizer_model = CountVectorizer(ngram_range=(1, 3), stop_words=list_of_stop_words) topic_model.update_topics(docs, vectorizer_model=vectorizer_model)

These are the topics generated from the subset of contractual clauses that were classified

topics_df=topic_model.get_topic_info() topics_df

Below is a representation of the frequency of occurrance of each of the three topics

This chart is empty

Chart was probably not set up properly in the notebook

The following words occur most frequently in all contracts

These words occur in the second most frequent topic cluster -

Named Entity Recognition (using OpenNyAI)

To see what statutes occur most frequently (alongside other indicators such as Organisation, Judge, Lawyer, and Geo Political Location) one can utilise the fine-tuned LegalBert model implemented by OpenNyAI in their pipeline https://github.com/OpenNyAI/Opennyai

I use this to demonstrate below for a subset of ~100 decisions how many times certain popular enactments occur. Due to CPU constraints, a larger inference could not be drawn at this time, but will be done later!

## USING Levenshtein Distance from TheFuzz library to cluster together the 'similar' statutes ## E.g. "Indian Contract Act" and "Contract Act, 1872" are essentially the same reference ## This is not entirely error free at this moment, but represents a good starting point import pandas as pd df_NER = pd.read_csv('combinedNER.csv') df_STATUTE = df_NER.dropna(subset=['STATUTE']) values = df_STATUTE['STATUTE'].tolist() #values = [item for sublist in df_STATUTE['STATUTE'].tolist() for item in sublist] import ast value_list = [] for value_lists in values: value_list = value_list +(ast.literal_eval(value_lists)) #print(value_list) threshold = 60 from thefuzz import fuzz from collections import defaultdict # Create a dictionary to store the fuzzy matches fuzzy_matches = defaultdict(list) # Create a set to keep track of strings that have already been matched matched_strings = set() # Iterate over the input list for string1 in value_list: if string1 not in matched_strings: for string2 in value_list: if string2!= string1 and string2 not in matched_strings: # Calculate the fuzzy match score score = fuzz.ratio(string1, string2) # If the score is above the threshold, add the string to the dictionary if score > threshold: fuzzy_matches[string1].append(string2) matched_strings.add(string2) matched_strings.add(string1) # Print the dictionary #print(fuzzy_matches) #df_fuzzymatches = pd.DataFrame.from_dict(fuzzy_matches, orient='index')

## These are the various statutes that have been matched ## and identified as containing similar references in the entire list of matches print(fuzzy_matches.keys())

keys_to_include = ['Indian Contract Act', 'Arbitration (Protocol and Convention) Act, 1937', 'Specific Relief Act, 1963', 'Income Tax Act', 'Electricity Supply Act'] # Using dictionary comprehension to create a subset subset_dict = {key: fuzzy_matches[key] for key in keys_to_include if key in fuzzy_matches} list_dict_subset = [{'statute': key, 'count': len(value)} for key, value in subset_dict.items()] df_subset_dict_count = pd.DataFrame(list_dict_subset) #print(df_subset_dict_count)

The following is a visualisation of the number of times certain key statutes occur in a subset of the dataset

This chart is empty

Chart was probably not set up properly in the notebook

.css-15w88e5{color:var(--chakra-colors-fg-neutral-primary);font-weight:inherit;letter-spacing:-0.09px;}Welcome to the Data Analytics Platform for "Indian Contracts in Adjudicated Texts ('ICAT')"

The following is a list of search queries entered so far

The following table randomly lists 10 results from the database

The following table and visualisation show the year of the decision for a random selection of 100 judgments from all results

Experimental: Conducting Topic Modelling on a subset of contract clauses using BERTopic

These are the topics generated from the subset of contractual clauses that were classified

Below is a representation of the frequency of occurrance of each of the three topics

The following words occur most frequently in all contracts

These words occur in the second most frequent topic cluster -

Named Entity Recognition (using OpenNyAI)

The following is a visualisation of the number of times certain key statutes occur in a subset of the dataset

Welcome to the Data Analytics Platform for "Indian Contracts in Adjudicated Texts ('ICAT')"