Sentient Expression
Sentient Expression® leverages respondents’ webcams, making it a globally-scaled, efficient platform for investigating natural responses to your media.
Services: Web App, Back-End Architecture, System Development
Core Technologies: Angular, MySQL, Redis, AWS, CLMTracker, FFMpeg
Sentient Expression Platform
The Sentient Platform can precisely predict consumer behaviors towards an ad by measuring emotions. With a highly scalable and robust backbone, a survey can handle 100-200 requests per minute with a minimum infrastructure of 2 servers behind a load-balancer. The architecture is flexible enough to add more servers depending upon the load and remove it as well.
- Hosting
AWS (EC2, RDS, S3, CloudFront, DynamoDB, SQS)
- User Interface
Angular, Flask
- Face Detection
CLMTracker
- Transactional Database
Aurora MySQL
- Inference Models
DLAMI (Deep Learning Amazon Machine Image) with Anaconda virtual environment; Apache MXNet deep learning framework
- Image Processing
FFMpeg
- Caching
Redis
- Computer Vision
OpenCV
Our Involvement
Our team is tightly aligned with the evolution of Sentient’s technology and continuously stays abreast of current trends by reading industry publications, monitoring the industry’s environment and competitor blogs, as well as message boards. To help Sentient stay ahead of business demands, our team provided the following solutions:
- Face Detection & Tracking
The program employs a set of sequential steps to detect a face for the first time in the underlying video stream.
- Viola-Jones: The objective of this step is to detect a bounding box/window around the face (if any exists at all) in the image data.
- MOSSE Correlation Filter: This step involves the application of the MOSSE correlation filter to track (one time) the two eyes and the nose (facial components) on the detected face.
- Procrustes Analysis: This step involves performing the Procrustes Analysis to determine the scaling, rotation, translation along the X-axis and translation along the Y-axis that need to be removed from the detected face in order to roughly align it with the mean face.
- Initial Guess of Feature Points: This step involves guessing the initial X and Y-coordinates of the facial feature points. The clmtracker program tracks a face by tracking 71 facial feature points.
- Prep the Drawing Canvas: This step involves preparing the canvas to draw the frame content (the image data) to work on.
- SVM & Response Image for Each Patch: In this step, the program extracts a patch window of 11X11 size and converts pixel data of this patch image into grayscale. For each of the patch images, SVM is performed to obtain the corresponding response image. Response images are stored into an array henceforth called as responses.
- Optimization for Best Feature Point Positions: In this step, the response images obtained from the previous step are optimized to get the best positions for the feature points.
After successfully detecting a face and having the current positions of all the 71 feature points of that face, the parameters or weights — with each iteration, on the same frame or continuing with a new frame — is constantly updated and applied to obtain the new positions of those feature points to keep track of the face in the underlying video. Once the tracking process is completed, the recorded webcam video and tracking data get uploaded to AWS S3 service.
- Inference on Tracking Data
Machine Learning models consume and process the image frames to extract the inferred emotion from each frame. The complete data set is then aggregated based on the timestamp and saved to the database. These database tables are then used to generate tabular results in the control panel.
- Visualization of Results
Charts depicting the various emotions are drawn on top of the base video and changes as per the timeline of the video.
