You can also read Jonny’s full series on the future of AI here. If you’d like to discuss how AI could solve your legal tech problems, you can reach Jonny on LinkedIn or Get in Touch.
Video Transcript:
Unilever had a requirement to review around 18,000 contracts in relation to 20 business critical data points in the context of an M&A transaction. This is a fairly common use case and lots of organizations go through it. Typically and historically, in-house legal teams have looked to law firms to provide these services for them.
The timescales were really critical for Unilever and an early analysis suggested that it would take around 9,000 hours for a full human review of those contracts and Unilever actually only had three months. Ultimately a full human review would have required 19 FTE being dedicated to this, so it presented a real opportunity to leverage technology and think about how we could transform how these services are delivered.
Ensuring Accuracy
We utilised DocuSign Insights, which is one of the market-leading AI contract review platforms, and it was a technology which was already in Unilever’s tech stack, which was a great starting point. We jointly analysed required outcomes for Unilever and ultimately designed a hybrid approach layering human QA and validation over an AI extraction process.
In this case we were utilising AI and automated technology to undertake an initial review of each and every contract to automatically extract the in-scope data points. On top of that we layered a human review to validate all outputs and an additional senior lawyer quality check layer on top of that to quality assure a proportion of the outputs.
The main reason for doing that is to ensure accuracy. AI technology is getting really advanced and year on year the models are getting better and better, as the technology learns from more and more data and use cases that it sees. But it’s still not perfect, as humans aren’t either, and the whole purpose of layering on this human approach is to ensure the quality of that data.
Designing the solution
The first part of the design process wasn’t technology-led. So before we jumped to any technology there was a real focus on the business problem to solve; looking at what it is that Unilever were trying to achieve, what was the desired outcome, and how can we work together to deliver some strategic value above and beyond just delivering the literal requirements of the project.
We leveraged a lot of out-of-the-box policies and we also built some new custom extraction policies to meet the requirements. A lot of the leading legal AI products come pre-trained to some extent. This enables the user of those products to leverage the expertise and the experience of existing customers and data sets that the technology’s seen before. We can therefore automatically extract a whole host of data points out of the box. As you might expect in any given use case, many customers are looking for the same kind of data points and the same use cases come up again and again.
One of the key principles we applied at the start of this project is let’s not try and reinvent the wheel. There’s a lot of great technology out there like DocuSign Insights, so let’s leverage what exists and make our job as easy as possible.
Training the Algorithm
Contracting parties is one of the really common data points that we look to leverage AI for, but whilst it’s very easy for a human to look at the front page of a contract or the recitals to work out who the parties are, those data points can be quite difficult to train the technology to look for. Quite often we see the party extractions out of the box typically perform at between 50 and 70 percent accurate. So at the start of this project, we built out a list of all relevant Unilever entities and then we built some custom logic into the platform which layered in the list of Unilever entities so that the AI model was then looking for a specific list in the context of the wider party review. That saw us increase the accuracy on that data point from around 55 percent to 98 percent.
Speed and Agility
We delivered a saving of around 6,500 human hours on the project compared to a full manual review. However, as mentioned previously, the speed of the transaction itself was key here – we found that we were about 70% quicker to complete the review working in this way than we could have achieved with a full human review.
We were able to review over 15,000 documents in less than one calendar month and this method of using AI to do the heavy lifting meant that we could review 20 times more documents than in a manual review. Typically, in a manual review if time is of the essence and resource is limited, most organizations would try to reduce the scope of the contract review and only look at material contracts. You’re therefore reducing the scope of what you might find in the process. Using AI technology in this way significantly reduced risk for Unilever because we were able to look at a much wider corpus of documents, at a similar cost, in less time.
The ability and the flexibility to increase the scope throughout the project was also a benefit. We started with 20 data points at the outset, but four weeks into the process, following some other conversations that were going on around the wider transaction, we recognized the need to extract some data related to an additional two data points. If that was a full manual review, you’d have to go all the way back to phase one and start to review all those contracts again manually, which just wouldn’t have been feasible in these timeframes. Instead, we were able to build a model – within a day – for each of those additional data points and re-run that again across all the contracts in a matter of a couple of days.
Validating the Accuracy of the Data
We did some testing to ascertain the accuracy of the data. One of the common misconceptions around AI is that it’s not accurate enough – and that’s one of the common challenges from lawyers. Interestingly – and bad news to a lot of lawyers – humans aren’t perfect either and we did a little bit of sampling on this project and actually found that the methodology that we delivered here saw an 18% improvement in the accuracy of data than if we’d have just used humans to have reviewed this data.
Jonny is an expert in delivering and leading legal innovation and automation projects, having previously led the innovation function at global law firm DWF. Jonny leads our data services practice and has a track record of helping clients to enable insight-driven decision making from their data.
Jonny has supported FTSE 100 companies with the implementation of AI technologies and has supported one of the largest global legal technology companies with the development of AI models for its market-leading platform. A lawyer by background, Jonny leads a team of highly skilled legal engineers operating in multiple jurisdictions.
