This chapter has presented two abilities to make your robot more interactive. To talk to your robot and have it speak to you makes it become alive with personal characteristics. The more voice commands or phrases it responds to and the more phrases your robot can speak, the more personality it will have.
An additional customization would be to interact with your robot face-to-face. Having your robot turn when spoken to and having him find the face of the speaker will add an air of awareness. The capability to detect a face would be an advantage.
There are two primary methods available in ROS to detect faces from a camera image stream. The first method is using a cascade classifier with Haar or LBP-like features. This relies on a method provided by OpenCV. The second method uses Eigenfaces to recognize faces in an image. Descriptions of these two methods are provided in the next two sections. The best approach to test these methods is to begin by using a USB webcam to stream the video images.
With OpenCV fully integrated with ROS, the algorithms and functions available from this library for robotics makes adding a new capability, such as face detection, extremely easy. The OpenCV algorithms have been designed to perform well and also refined for speed, efficiency, and accuracy.
The OpenCV cascade
classifier has two stages of implementation: training and detection. In the training stage, OpenCV provides the opencv_traincascade application to detect features in the images for classification. Two types of features can be classified by opencv_traincascade: Haar or
Local Binary Patterns (LBPs). Datasets generated by either method can provide good quality data for classification, but the LBP features are processed much more quickly. The LBP features are handled as integer values, which make processing LBP data several times faster than processing Haar data.
The OpenCV User Guide provides an in-depth understanding of the cascade classifier and instructions on how to prepare training data and run the training software; refer to http://docs.opencv.org/2.4/doc/user_guide/ug_traincascade.html.
The training software generates an .xml classifier file with the classified features. For faces, it is interesting to know that a training set of hundreds or even thousands of sample faces need to be included in the dataset of positive samples. The dataset should contain samples of all race and age groups as well as facial hair variations and facial emotions. These positive samples should be .png files and the face or facial features should be marked with bounding shapes. The opencv_createsamples utility is used to generate this image set into vector (vec) format for training. A set of negative images is also needed that do not contain any faces. With these two datasets, the opencv_traincascade application can be used to train a cascade classifier.
Already prepared datasets are available in the OpenCV GitHub repository. Haar cascade classifier files exist for detecting a face from the front and also from the side. Files with facial feature detection, such as eyes and smiles, are also available. A sample vector file trainingfaces_24-24.vec is offered for training a face detector. These files can be found at https://github.com/Itseez/opencv/tree/master/data.
To understand how the cascade classifier in OpenCV works, refer to the tutorial at http://docs.opencv.org/2.4/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html#cascade-classifier.
This tutorial uses the haarcascade_frontalface_alt.xml and haarcascade_eye_tree_eyglasses.xml files from the GitHub repository.
The API for cascade classifier object detection is found at http://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html.
The objdetect module in OpenCV handles the detection of objects after they have been classified in the training stage of the cascade classifier. This object detection method was proposed by Paul Viola and Michael Jones and improved upon by Rainer Lienhart. An explanation of the classifier is provided at the website.
A good example of implementing an OpenCV Haar cascade face detector can be found in the book ROS by Example-Indigo (Volume 1) by Patrick Goebel. You will be shown how a cascade classifier is used and the software is available to adapt to your own application.
The ROS package face_detector offers an improvement on the OpenCV face detection method provided by the Haar-like cascade classification described in the previous section. This package is designed to work with a stereo camera or a 3D sensor, such as Kinect, ASUS Xtion, or PrimeSense. The original set of face detections obtained from the cascade classifier is processed a second time using the depth information from the images. Each positive face image in the dataset is evaluated based on the depth information to determine whether the size of the face is realistic for a human. If so, the face is preserved in the dataset. Many false positives are pruned from the dataset using this method.
The face_detector package can be implemented in two ways: on a continuous image stream or as an action. The action method allows you to start face detection at any time, and the process runs independently until at least one face is found.
To implement this ROS package on your robot, visit http://wiki.ros.org/face_detector.
The use of a stereo camera or a 3D sensor is necessary.
Still, this method of face detection is inferior when based on just the detection of geometric features within a face. Research has shown that a more robust approach is needed to increase the likelihood that face recognition will be accurate. Methods such as Eigenfaces and Fisherfaces have been advanced to adopt a more holistic approach to face recognition. These methods have been incorporated into a new class of objects performing face recognition in OpenCV.
The new FaceRecognizer class in OpenCV implements new technology in development for face recognition. The presentation and analysis of the algorithms used for this class is not the subject of this book, but we will present the available options for this software in order for you to implement them on your robot projects. The algorithms available for the FaceRecognizer class are as follows:
- Eigenfaces (via
createEigenFaceRecognizer()) - Fisherfaces (via
createFish - Binary Pattern Histograms (via
createLBPHFaceRecognizer())
Refer to the OpenCV tutorial on Face Recognition found at http://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html.
A detailed explanation of these face recognition algorithms is provided. Instructions on preparing the datasets are also given. Face datasets for this FaceRecognizer class reference the databases found at http://face-rec.org/databases/.
The FaceRecognizer class is described at http://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_api.html.
To explore more of OpenCV's face recognition capabilities, such as gender classification, check out the full list of documentation at http://docs.opencv.org/2.4/modules/contrib/doc/facerec/index.html.
The ROS metapackage cob_people_perception was developed by the Fraunhofer Institute for Manufacturing Engineering and Automation for their Care-o-bot research platform (http://www.care-o-bot.de/en/research.html). This metapackage contains several packages for detecting and recognizing faces as well as detecting and tracking humans with a 3D sensor or laser scanner. The cob_people_detection package performs head and face detection utilizing a Viola-Jones classifier. The second step employs face identification techniques based on either Eigenfaces or Fisherfaces to perform the complete identification of known faces. The tracking of these faces increases confidence in the results.
The first time the people_detection client node is run, no identification data exists on the computer. The node can be run either in manual or automatic mode, but images captured should only have a single face present in the image. In automatic mode, images of an individual should be captured between 40 and 100 times varying poses and lighting conditions. It is also important to capture the images of more than one person during the first session of data gathering. Detected faces should be tagged with an identification label. When the training is complete, a recognition model should be build from the training set data.
Complete instructions for installing the cob_people_detection package, creating the database, and using the software is found at http://wiki.ros.org/cob_people_detection.
The GitHub repository for cob_people_perception and this package is found at https://github.com/ipa-rmb/cob_people_perception.
The ROS package face_recognition for the recognition of faces in a video image stream was developed in the Robolab at the University of Luxembourg. This package was developed by Shervin Emami and uses the FaceRecognizer class provided by OpenCV. The face_recognition ROS package provides a simple actionlib server interface for performing different face recognition operations in an image stream.
Face recognition is performed using Eigenfaces and Principal Component Analysis (PCA) provided by the FaceRecognition methods. Emami describes the methods further on his website (http://www.shervinemami.info/faceRecognition.html).
Further information on this package can be found at http://wiki.ros.org/face_recognition.
The repository for the
face_recognition source code is available at https://github.com/procrob/face_recognition.
Shervin Emami has updated his face recognition techniques in his book, Mastering OpenCV with Practical Computer Vision Projects; refer to the Face Recognition using Eigenfaces or Fisherfaces section in Chapter 8. The source code for this chapter is freely available at https://github.com/MasteringOpenCV/code/tree/master/Chapter8_FaceRecognition.
To use this software, at least three face and eye detection .xml datasets will need to be downloaded from the OpenCV GitHub data repository; refer to https://github.com/Itseez/opencv/tree/master/data (link repeated from earlier).
With this variety of techniques for face detection and recognition, it will be a great advantage to have your robot recognize you and your friends!