Palm motion detection and tracking for drawing!
Recently I made a program to track my hand movements and translate that into cursor movements on the computer screen. I used OpenCV and Python3 to do so. Here’s how it works!
First the program reads the video from the connected camera (USB webcam in my case). After that, it converts the frame to black and white. The B/W image is then thresholded so that points that are brighter are considered as white and the darker points are considered black.
From here the program detects the largest contour in the image.
After that, I detect the convex hull in the image. With this, it will join the tips of the fingers to form a polygon. To better explain my point, here’s how it looks.
After finding the convex hull, to detect the pointer finger, we find the furthest point from the center that is within the convex hull (in blue).
Let’s call this point fPos(x, y). To improve the tracking, it is best to use a pointed object such as a pencil to test this program.
Now we need to find the position on the screen where the mouse cursor should be relative to fPos(x, y). I did this by first finding the ratios between the dimensions of the video. My webcam’s video resolution is 640x480 and my screen’s resolution is 1920x1080. So the center of the video will be videoCenter(320, 240) and the screen’s center will be screenCenter(960, 540). If we divide videoCenter’s x-coordinate by screenCenter’s x-coordinate, we get ratioX. Similarly we can do the same with videoCenter and screenCenter’s y-coordinates to obtain ratioY. Now to get the position of the cursor on the screen relative to fPos is:-
sPos(x1, y1) where x1 = (x*ratioX)÷2 and y1 = (y*ratioY)÷2
Now using the python module pynput we can set the mouse cursor position to sPos and now we can control the mouse with hand movements.
Conclusion
Though the program has a few bugs now, I had fun doing it and I hope that others can have fun with it too. If you’d like to test this program, just have Python and OpenCV installed along with a few other modules listed here -
You’d also need to clone the project from github here and run main-1.py. Controls -
d - drawing, e - eraser, left-click - clear screen, right-click - toggle mouse control
That’s all I have for now!