Using Cognitive Services to locate faces in images
Microsoft offers Cognitive Services, which helps developers to leverage AI features in their applications.
In this recipe, you'll learn how to use the Computer Vision API (Cognitive Service) to detect faces within an image. We will be locating faces, capturing their coordinates, and saving them in different areas of Azure Table storage based on gender.
Cognitive Services apply AI algorithms, so they might not always be accurate. The accuracy returned by Cognitive Services is always between 0 and 1, where 1 means 100% accurate. You can always use the accuracy value returned by Cognitive Services and implement your custom requirements based on the accuracy.
Getting ready
To get started, we need to create a Computer Vision API and configure its API keys so that Azure Functions (or any other program) can access it programmatically.
Make sure that you have Azure Storage Explorer installed and configured to access the storage account that is used to upload the blobs.
Creating a new Computer Vision API account
In this section, we'll create a new Computer Vision API account by performing the following steps:
- Create a function app, if one has not been created already, by choosing .NET Core as the runtime stack.
- Search for Computer vision and click on Create.
- The next step is to provide all the details (name, resource group, and subscription) to create a Computer Vision API account. At the time of writing, the Computer Vision API has two pricing tiers. For this recipe, select the free one, F0, which allows 20 API calls per minute and is limited to 5,000 calls each month. For your production requirements, you should select the premium instance, S1.
Having created the Computer Vision API account, we'll now move on to configure the application settings.
Configuring application settings
In this section, we'll configure the application settings of Azure Functions by performing the following steps:
- Once the Computer Vision API account has been generated, navigate to the Keys and Endpoint blade and copy KEY 1 into the notepad:
Figure 3.1: Computer Vision keys
- Navigate to your Azure Functions app, configure Application settings with the name Vision_API_Subscription_Key, and use any of the preceding keys as its value. This key will be used by the Azure Functions runtime to connect to and consume the Computer Vision Cognitive Services API.
- Make a note of the location where you are creating the Computer Vision service. In this case, it is East US. In terms of passing the images to the Cognitive Services API, it is important to ensure that the endpoint of the API starts with the location name. It would be something like this: https://eastus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Faces&language=en
Let's now move on to the next section to learn how to develop the Azure function.
How to do it…
In this section, you are going to learn how to leverage Cognitive Services in the blob trigger by performing the following steps:
- Create a new function using one of the default templates named Azure Blob Storage Trigger.
- Next, provide the name of the Azure function along with the path and storage account connection. We will upload a picture to the Azure Blob Storage trigger (image) container (mentioned in the Path parameter in Figure 3.2) at the end of this section:
Figure 3.2: Creating an Azure Blob storage trigger
Note
While creating the function, the template creates one blob storage table output binding and allows you to provide a name for the Table name parameter. However, you can't assign the name of the parameter while creating the function. You will only be able to change it after it has been created. After reviewing all the details, click on the Create button to create the Azure function.
- Once the function has been created, navigate to the Integrate tab, click on New Output, choose Azure Table Storage, and then click on the Select button. Provide the parameter values and then click on the Save button, as shown in Figure 3.3:
Figure 3.3: Azure Table storage output bindings
- Let's now create another Azure Table Storage output binding to store all the information for women by clicking on the New Output button in the Integrate tab, selecting Azure Table Storage, and then clicking on the Select button. This is how it looks after providing the input values:
Figure 3.4: Azure Table storage output bindings
- Once you have reviewed all the details, click on the Save button to create the Azure Table Storage output binding and store the details pertaining to women.
- Navigate to the code editor of the Run method and copy the following code. The code will collect the image stream uploaded to the blob, which will then be passed as an input to Cognitive Services, which will then return some JSON with all the face information, including coordinates and gender details. Once this face information is received, you can store the face coordinates in the respective table storage using the table output bindings:
#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"
using Newtonsoft.Json;
using Microsoft.WindowsAzure.Storage.Table; using System.IO;
using System.Net; using System.Net.Http;
using System.Net.Http.Headers;
public static async Task Run(Stream myBlob,
string name,
IAsyncCollector<FaceRectangle> outMaleTable,
IAsyncCollector<FaceRectangle> outFemaleTable,
ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
string result = await CallVisionAPI(myBlob); log.LogInformation(result);
if (String.IsNullOrEmpty(result))
{
return;
}
ImageData imageData = JsonConvert.DeserializeObject<ImageData>(result);
foreach (Face face in imageData.Faces)
{
var faceRectangle = face.FaceRectangle;
faceRectangle.RowKey = Guid.NewGuid().ToString();
faceRectangle.PartitionKey = "Functions";
faceRectangle.ImageFile = name + ".jpg";
if(face.Gender=="Female")
{
await outFemaleTable.AddAsync(faceRectangle);
}
else
{
await outMaleTable.AddAsync(faceRectangle);
}
}
}
static async Task<string> CallVisionAPI(Stream image)
{
using (var client = new HttpClient())
{
var content = new StreamContent(image);
var url ="https://<location>.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Faces&language=en";
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", Environment.GetEnvironmentVariable("Vision_API_Subscription_Key"));
content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
var httpResponse = await client.PostAsync(url, content);
if (httpResponse.StatusCode == HttpStatusCode.OK)
{
return await httpResponse.Content.ReadAsStringAsync();
}
}
return null;
}
public class ImageData
{
public List<Face> Faces { get; set; }
}
public class Face
{
public int Age { get; set; }
public string Gender { get; set; }
public FaceRectangle FaceRectangle { get; set; }
}
public class FaceRectangle : TableEntity
{
public string ImageFile { get; set; }
public int Left { get; set; }
public int Top { get; set; }
public int Width { get; set; }
public int Height { get; set; }
}
- The code has a condition to check the gender and, based on the gender, it stores the information in the respective table storage.
- Create a new blob container named images using Azure Storage Explorer, as shown in Figure 3.5:
Figure 3.5: Azure Storage—Create Blob Container
- Let's now upload a picture with male and female faces to the container named images using Azure Storage Explorer, as shown in Figure 3.6:
Figure 3.6: Azure Storage—Create Blob Container—Upload Files
- The function will get triggered as soon as you upload an image. This is the JSON that was logged in the Logs console of the function:
{
"requestId":"483566bc-7d4d-45c1-87e2-6f894aaa4c29", "metadata":{ },
"faces":[
{
"age":31, "gender":"Female",
"faceRectangle":{
"left":535,
"top":182,
"width":165,
"height":165
}
},
{
"age":33,
"gender":"Male",
"faceRectangle":{ "left":373,
"top":182,
"width":161,
"height":161
}
}
]
}
Note
A front-end developer with expertise in HTML5 and canvas-related technologies can even draw squares that locate the faces in images using the information provided by Cognitive Services.
- The function has also created two different Azure Table storage tables, as shown in Figure 3.7:
Figure 3.7: Azure Table storage—output values of the cognitive services
Note
The APIs aren't 100% accurate in identifying the correct gender. So, in your production environments, you should have a fallback mechanism to handle such situations.
There's more...
The face locator templates invoke the API call by passing the visualFeatures=Faces parameter returns information relating to the following:
- Age
- Gender
- Coordinates of the faces in the picture
Note
Learn more about the Computer Vision API at https://docs.microsoft.com/azure/cognitive-services/computer-vision/home.
Use the Environment.GetEnvironmentVariable("KeyName") function to retrieve the information stored in the application settings. In this case, the CallVisionAPI method uses the function to retrieve the key, which is essential for making a request to Microsoft Cognitive Services.
Note
It's considered a best practice to store all the keys and other sensitive information in the application settings.
In this recipe, you have learned how to integrate Cognitive Services with Azure Functions. Let's now move on to the next recipe to learn how to integrate Azure Functions with Logic Apps.