Recognising Fingertips

Pulling an all nighter is always rewarding , so Friday night I made my first attempt at detecting the fingertips on the hand. Most people use the "convex irregularities" method. I really didn't like that method. It seems sloppy and doesn't detect all fingers. I prefer the kcosines method as described in "Vision-Based Finger Action Recognition by Angle Detection and Contour Analysis".

These are the results of my first attempt.

This week I'll concentrate on making tracking and contour extraction more robust because as you can see at some points the contours break up. I guess selecting my wooden office desk as a testing area proved to be quite a challenge.

Extracting and stabilizing contours

It's been a busy week. I'm now at the stage of contour extraction. Using my adaptive skin classifier and the samples gathered from the detector I build a histogram model for the hand and extract the contours around it. As you can see it is quite robust and works under different illumination. All these are possible with the assumption that the detector doesn't return a false positive. While HAAR cascades are fairly good at the job they don't have a 0% percent of false positives so I intend to add a Fourier hand validator.

Using train cascades

The detector I trained back in November doesn't work for me anymore, so I'm attempting to train another one using the train cascades executable provided in the OpenCV framework.

I gathered 70k negatives , 30k positives and I run exactly the same code and boom crash burn. All I got was the message "Train dataset for temp stage can not be filled. Branch training terminated."

After 2 hours of screwing around all possible combinations and reading the same problems on the net I decided to look at the code (damn messy C++ code). After that I decided to reduce the number of negatives to the same number of positives and voila it worked.

I've come to believe that OpenCV was originally written by a couple of uber programmers and then left to be wrecked by drunken monkeys.

Building an adaptive skin classifier

Building an adaptive skin classifier is quite some work.
I've seen some examples on the net. The good ones are not real time and the ones that are  simply don't cut it.
A small lighting variation, a slightly complex background and the classifier is lost.

Many examples I've seen use hard coded variables. That is obviously wrong.
I've also seen many that use the RGB color space which is also wrong.
I strongly recommend the HSV color space because it is slightly more lighting invariant or at least normalized RGB.

I also recommend using more features than the color channels. You also have to experiment with different bin sizes in order to get real time performance.

I recently found out that arithmetic accuracy is also important because of all the normalization operations.

During segmentation is important to postpone thresholding  because of the mass loss of information. I prefer using the probability map during tracking and later threshold to extract the contours.

Well I'm getting closer...

 

I'm usually very critical of the stuff I make but today I feel quite satisfied with the results.

The constraints of histogram tracking

While in the last months I have implemented and tested a couple of histogram based tracking algorithms I only recently realized their inherent constraints. If you use it for hand tracking and you have your arms naked or your face exposed it is very easy for the tracker to get confused because of the coarse quantization of the histograms.
The worst part is that there is little you can do :

• I tried using a different color space such as HSV. While it is far better that RGB for tracking skin and coping for small lighting variances it is still not enough and the tracker often gets confused especially when the hand goes out of view.

• I tried incorporating different features such as edge magnitude (bad idea) and edge orientation (much better) and it had the effect of better localizing the detector. 

Integral histograms are still an option but there are far too slow for a real time application like the one I'm working on.

Adaptive Classifier Results

Below are the first results of the adaptive skin classifier I implemented.
There is no prior knowledge, the skin histogram model is learned from the HAAR detector regions and some morphological filtering.

 There is an awful lot of parameters to tweak, it's getting ridiculous.
Currently I'm trying to incorporate the adaptive mask to the particle filter but there are some technical issues.
Anyway, the tracking is being handled by the mean shift tracker which sucks.

Mixing Histograms

In the last few days I implemented an adaptive skin classifier in make it easier for the tracking algorithms to find the objects. Emgu already has an adaptive skin classifier but you cannot change any parameters and the results are horrible.

Anyway, mixing two histograms in Emgu is not as straightforward as one might think.
The internal representation of the histograms is the N-th dimensional matrix class , MatND and the OpenCV does not overload any operators.

Thankfully there is a function in the cvInvoke that does the job.
I wasted about half an hour trying to use the cvAdd function but it crashed the whole thing about a dozen times, so I ended up using the cvAddWeighted function.

Below is the code snippet for histogram mixing :

        /// <summary>
        /// Mixes two histograms , new histogram is _mixCoeff * _histTarget + (1 - _mixCoeff) * _histMix
        /// Old target histogram is destroyed
        /// </summary>
        /// <param name="_histTarget">Target histogram</param>
        /// <param name="_hist1">Histogram 1</param>
        /// <param name="_hist2">Histogram 2</param>
        /// <param name="_mixCoeff">Mixing Coefficient</param>
        public static void MixHistograms(
            DenseHistogram _histTarget,
            DenseHistogram _hist1,
            DenseHistogram _hist2,
            float _mixCoeff)
        {
            CvInvoke.cvAddWeighted(
                _hist1.MatND.Ptr,
                _mixCoeff,
                _hist2.MatND.Ptr,
                1.0d - _mixCoeff,
                0,
                _histTarget.MatND.Ptr);
        }

PS. On a following post I will post the adaptive skin classifier.

Image Segmentation

In computer vision, segmentation refers to the process of partitioning a digital image into multiple segments. The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.

I have a dataset of hands against a background. As you can see the background is not uniformly lit.

In addition to that the hand's position ,scale and rotation is not static.
I don't have the ground truth of the hand and I don't want to waste my twenties marking it myself.
It is fairly obvious that thresholding simply doesn't cut it and given that I don't have any skin information to begin with what can I do?

Well , the only thing you can do is improvise.

My first thought was to use K-Means to try to split the image into two parts : a background class containing the backdrop and a foreground class containing the hand pixels.

I tested that a bit and found that it worked pretty well. So I decided to upgrade it and use Expectation Maximization using Gaussian Mixtures.
A subproblem is given the two classes is discriminating between the skin and background classes. After the mask is computer it is advisable to do some post-processing to remove some rogue pixels.

Below I include the code to do all the above :

%------------------------------------------------------------------------
        % gather features
        [rows cols colors] = size(imgDouble);
        [X,Y] = meshgrid(0 :1/(rows-1) : 1,0 :1/(cols-1) :1);
        X = reshape(X,rows*cols,1);
        Y = reshape(Y,rows*cols,1);
        imgFeatures = [X Y];

        for i = 1 : colors
            imgMasked(:,:,i) = medfilt2(imgMasked(:,:,i),[5 5],'symmetric');
            imgFeatures = [imgFeatures reshape(imgDouble(:,:,i),rows*cols,1)];
        end
%------------------------------------------------------------------------
        % use EM with gaussian mixture
        objGM = gmdistribution.fit(imgFeatures,noClasses,'Replicates',10);
        % find which of the classes is the hand
        Sigma = zeros(noClasses,1);
        for i = 1 : noClasses
            Sigma(i) = sum(sum(objGM.Sigma(:,:,i)));
        end
        [C,I] = max(Sigma);
        skinClass = I(1);
        % use gaussian mixtures to classify pixel features
        [IDX,nlogl,P] = cluster(objGM,imgFeatures);
        % fill holes to create a mask
        IDXimg = reshape(IDX,rows,cols);
        IDXimg = IDXimg == skinClass;
        IDXimg = bwmorph(IDXimg,'erode',1);
        IDXimg = bwareaopen(IDXimg, 100 / scale);
        imgMasked(repmat(IDXimg == 0,[1 1 colors])) = 0;
%------------------------------------------------------------------------

I used a bit of active contours cleaning that I don't include. Below I include some of the resulting images which are very satisfying and can be used to bootstrap any system for even better results :