In the last chapter, we covered how AWS AI services such as Amazon Textract and Amazon Comprehend Medical can help automate healthcare workflows quickly. In this chapter, we will talk about why we need humans in the loop in document processing workflows, and how setting up human-in-the-loop (HITL) processes with Amazon Augmented AI (Amazon A2I) can help improve accuracy in your existing document processing workflows with Amazon Textract.
Amazon A2I is a managed service that makes it easy to build the workflows required for the human review of machine learning (ML) predictions. In this chapter, we will cover the following topics:
For this chapter, you will need access to an AWS Account. Please make sure to follow the instructions specified in the Technical requirements section in Chapter 2, Introducing Amazon Textract, to create your AWS Account, and log in to the AWS Management Console before trying the steps in Improving the accuracy of document processing workflows.
The Python code and sample datasets for our solution can be found at the following link: https://github.com/PacktPublishing/Natural-Language-Processing-with-AWS-AI-Services/tree/main/Chapter%2013 .
Check out the following video to see the Code in Action at https://bit.ly/3jBxBQq.
Please use the instructions in the following sections along with the code in the repository to build the solution.
In the previous chapters, we have discussed how you can use Amazon Textract and Amazon Comprehend to automate your existing document processing workflows with AWS AI services. We covered some of the key use cases such as using Comprehend to analyze SEC filing reports and using Textract to extract text from any document or quickly digitize any document. We also spoke about how these AI services provide a confidence score with each predicted text, word, line, or entity. Now, the questions that customers often ask are about how to improve these predictions and to make sure they are highly accurate. In most AI systems, it's either AI doing the automation process or it's only humans or manual processes.
The ideal scenario would be both humans and AI working together so that the results predicted by these AI systems can be reviewed by humans to make sure they are highly accurate. This applies to scenarios where we are processing highly critical information in our document processing workflows, for example, processing loan applications, invoice processing, mortgage processing, or legal documents reviews, and so on. In all of these scenarios, you would want a human to validate or review whether the loan amount or invoice amount predicted by the ML or AI systems was accurate because if it was not, one zero could change your life! It could lead to millions in losses for the organization. Also, in the case of low-confidence predictions, you would also want to have the AI-predicted information checked over by a human reviewer. For example, in the case of claims processing, if your ML system has predicted a low confidence score for a claim, you would want to reject or approve the claim based on a human review. This working of humans and AI/ML together builds trust with the process and it also reduces the time to market for the overall document processing solution.
In this section, we covered why you would need to set up a human review loop with your existing document processing workflows – that is, it helps to improve accuracy and build trust with AI systems. In the next section, we will talk about some of the key benefits of using Amazon A2I when setting up HITL workflows.
Amazon A2I is a managed service that makes it easy to build and manage human reviews with any ML application. It integrates directly with Amazon Textract and Amazon Rekognition to set up human workflows for use cases such as content moderation or document processing. You can also create custom workflows with any Sagemaker model. Some of the benefits of using Amazon A2I are as follows:
We have discussed some of the benefits, now, let's deep dive into how Amazon A2I works with any AWS AI service, your custom model, or Sagemaker-deployed models through the following architecture:
Let's understand the diagram by going through each step:
From Amazon S3, your corrected or human augmented/validated AI/ML predictions can be sent to client applications.
Note
With Amazon Textract, you can only augment or validate your predictions. The model retraining feature is not supported yet, as Amazon Textract does not support custom models. However, with Amazon Comprehend's custom classification and the custom entity you can use Amazon A2I to retrain your custom models and improve accuracy.
In this section, we covered the benefits of using Amazon A2I and saw how it works to improve the accuracy of your AI/ML system. In the next section, we will walk you through how you can set up HITL workflows with document processing workflows using Amazon Textract and Amazon A2I.
We covered how A2I works in general with any AWS AI service using an architecture. In this section, we will talk specifically about how Amazon A2I integrates with Amazon Textract so that you can automate your existing document workflows with a HITL. We will also show you a walkthrough on how to set up a human review workflow using the AWS Console.
Let's start by walking through how Amazon A2I works with Amazon Textract:
Your scanned documents are sent to Amazon Textract. Amazon Textract extracts text, key-value pairs, and tables from the document. Now for this extracted text, key-value pairs, and tables, you also get confidence scores. You can set up a threshold that Amazon A2I triggers to create human review loops. You can set up a threshold with Amazon Textract for three things:
Note
Amazon Textract and Amazon A2I have a native integration in the API only for the key-value or form data extraction should you want to set up human reviews for text or tabular data detected by Amazon Textract.
You can use a custom A2I UI template and set up a loop using the starthumanloop A2I API rather than the Textract API. We will add examples on how you can set up a custom loop (in the Further reading section) when Amazon A2I starts a human loop API: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-start-human-loop.html#a2i-instructions-starthumanloop.
Let's go back to our fictitious company use case. This bank, LiveRight, wants to digitize its check processing system. The current system is a manual review of the check amount and date, which is time-consuming and error-prone. We will show you how LiveRight can use Amazon Textract and Amazon A2I to automate their current check processing system. We will use this sample check document:
We will use the check to validate the dollar value and date to make sure it's accurate before we issue it to the payee. To do this, we will walk through the following steps:
So, let's get started!
First, we will talk about how you can create an Amazon S3 bucket and upload the sample document for Textract A2I processing. Amazon A2I needs an Amazon S3 bucket to store the human-annotated results, as we covered in the architecture in Figure 13.1. We will also enable the Cross-Origin Resource Sharing (CORS) configuration needed to set up the A2I loop in this S3 bucket.
This configuration allows the A2I browser permission to download resources from this S3 bucket. To do this, follow the next steps:
Then, create a bucket with a2i-demos as the bucket name, using the instructions in Chapter 2, Introducing Amazon Textract.
[
{
"AllowedHeaders": [],
"AllowedMethods": [
"GET"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
In this section, we covered creating an S3 bucket, uploading the sample check, and setting up the CORS configuration. Now, we will set up a private work team.
In this section, we will walk you through how to create a private work team and add yourself as a worker using your email:
You will receive an email after adding a worker to the private team you created. This email will have a link to sign in to your A2I task UI portal, along with your username and password.
In this section, we covered how you can create a private workforce. Now, let's move on to the next section to set up a human review workflow.
In this section, we will show you how you can create a human review workflow with Amazon Textract. You need the Amazon S3 bucket and private workforce created in previous sections, and a UI where the reviews and options ot set the threshold conditions for triggering the human review loop will appear:
In the Workflow settings screen, enter Textract-check in the Name field. In S3 bucket enter the name of the bucket you created: <s3://a2i-demos/> . For IAM role, click Create new role and choose any S3 bucket. Your settings should look like following:
We set up two out of three trigger conditions for Textract A2I to trigger a review. The human loops will only be created when the preceding conditions are met – that is, if any key is missing, such as dollar amount and date, or any form key has a confidence score less than 90%.
In this section, we showed how you can create a flow definition for the human review loop. In the next section, we will show you how you can trigger this flow definition in the Textract Analyze document API while passing documents.
In this section, we will show you how you can trigger the A2I flow definition you just created in the Textract Analyze document API with forms.
If you have not done so in the previous chapters, you will first have to create an Amazon SageMaker Jupyter Notebook and set up Identity and Access Management (IAM) permissions for that notebook role to access the AWS services we will use in this notebook. After that, you will need to clone the GitHub repository (https://github.com/PacktPublishing/Natural-Language-Processing-with-AWS-AI-Services), go to the Chapter 13 folder, and open the chapter13 Improving accuracy of document processing .ipynb notebook:
Note
Make sure the IAM role in the notebook has permissions for AmazonTextractFullAccess (https://console.aws.amazon.com/iam/home?#/policies/arn:aws:iam::aws:policy/AmazonTextractFullAccess$jsonEditor) and AmazonAugmentedAIFullAccess (https://console.aws.amazon.com/iam/home?#/policies/arn%3Aaws%3Aiam%3A%3Aaws%3Apolicy%2FAmazonAugmentedAIFullAccess).
bucket="<your s3 bucket name>"
humanLoopConfig = {
'FlowDefinitionArn':"<enter flow definition arn created> ",
'HumanLoopName':"textract-10",
'DataAttributes': { 'ContentClassifiers': [ 'FreeOfPersonallyIdentifiableInformation' ]}
}
response = textract.analyze_document(
Document={'S3Object': {'Bucket': bucket,'Name': "samplecheck.PNG"}},
FeatureTypes=["FORMS"],
HumanLoopConfig=humanLoopConfig
)
Copy the workteam ARN and paste it in the notebook cell as follows:
WORKTEAM_ARN= "enter your private workteam arn"
workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]
for resp in completed_human_loops:
splitted_string = re.split('s3://' + bucket + '/', resp['HumanLoopOutput']['OutputS3Uri'])
output_bucket_key = splitted_string[1]
print(output_bucket_key)
response = s3.get_object(Bucket= bucket, Key=output_bucket_key)
content = response["Body"].read()
json_output = json.loads(content)
pp.pprint(json_output)
print(' ')
You will get the output with the human-annotated answers in the following screenshot:
In this section, we gave you a notebook walkthrough of invoking the Textract Analyze document API with a HITL following definition configuration for key-value pair or Forms detection.
In this chapter, we covered how you can improve the accuracy of your current document processing workflows when using Amazon Textract for automating these document workflows. We introduced Amazon A2I and how it can help improve the accuracy of your text predictions and can be integrated with Amazon Textract to set up a human review workflow. We also discussed how A2I can be used for model retraining with custom models to improve accuracy, which we will cover in the next chapter.
To further demonstrate how A2I works in action, we have also provided a 10-minute YouTube video tutorial by one of the authors cited in the Further reading section.