Gesture controlled desktop using webcam

Download control - 3. NET wrapper for openCV, to perform image processing, through which we try to recognize hand gestures and control mouse using these gestures.

Gesture controlled desktop using webcam

With the TensorFlow object detection api, we have seen examples where models are trained to detect custom objects in images e. Naturally, an interesting next step is to explore how these models can be deployed in real world use cases — for example, interaction design.

In this post, Gesture controlled desktop using webcam cover a basic body-as-input interaction example where real time results from a hand tracking model web cam stream as input is mapped to the controls of a web-based game Skyfall.

The system demonstrates how the integration of a fairly accurate, light weight hand detection model can be used to track player hands and enable realtime body-as-input interactions. Want to try it out? Project code is available on Github.

Importantly, appropriating parts of the human body for gesture based interaction has been shown to improve user experience [2] and overall engagement [1].

While the idea of body as input is not entirely new, existing approaches which leverage computer vision, wearables and sensors kinect, wii, [5] etc sometimes suffer from accuracy challenges, are not always portable and can be challenging to integrate with 3rd party software.

Advances in light-weight deep neural networks DNNsspecifically models for object detection see [3] and key point extraction see [4] hold promise in addressing these issues and furthering the goal of always available body as input. These models allow us track the human body with good accuracy using 2D images and with the benefit of easy integration with a range of applications and devices desktop, web, mobile.

While tracking from 2D images does not give us much depth information, it is still surprisingly valuable in building interactions as shown in the Skyfall game example.

Key features of this free Windows mouse gesture control are:

Body as input on a large display. Game Mechanics Skyfall is a simple web based game created using planck. The play mechanism for SkyFall is simple. Players earn points by moving a paddle to catch the good balls white and green balls and avoid bad balls red balls.

In the example below, the player can control the paddle by moving the mouse or by touch drag on a mobile device. Try out a mouse controlled version of SkyFall: Catch good balls white, greenavoid bad balls red by moving the mouse.

Basic Usage

Press space bar to pause game and key s to toggle sound. View on Codepen here. This is a pretty simple and fun game. However, we can make it even more engaging by allowing the user control the paddle using their body hand. The goal is to accomplish this by detecting hand position using web cam video stream— no additional sensors or wearables.

Adding Gesture Based Interaction To add gesture interaction, we replace the mouse controls above with a system that maps the movement of the players hand to the game paddle position.

In the current implementation, a python application app.

Gesture controlled desktop using webcam

Please see the blog post to learn more on how the hand tracking model is built. For any errors or issues related to loading the hand model, please see the hand tracking Github repo and issues.

This example follows a similar approach where a multi threaded python app reads web cam video feed and outputs bounding boxes for each hand detected. Note that hand detection is done on a frame-by-frame basis and the system does not automatically track hand across frames.

However, this type of inter-frame tracking is useful as it can enable multiple user interaction where we need to track a hand across frames think a bunch of friends waving their hands or some other common object, each controlling their own paddle.

To this end, the current implementation includes naive euclidean distance based tracking where hands seen in similar positions across frames are assigned the same id. Once each hand in the frame is detected and a tracking id assignedthe hand coordinates are then sent to a web socket server which sends it out to connected clients.

A tracking id of 0 is assigned to one hand and 1 to the other. These are then mapped to the paddles on the game interface. Game Interface The game interface connects to the web socket server and listens for hand detection data.

Gesture controlled desktop using webcam

Each detected hand is used to generate a paddle, and the coordinate of the hand in the video frame is used to relatively position the paddle on the game screen. Realtime detected hand coordinates center of bounding boxes are mapped to paddle positions as controls.Use Facial Gestures and webcam To Control Your Windows PC With eViaCam.

Use Facial Gestures and webcam To Control Your Windows PC With eViaCam. and the square will move with your gesture. Nov 25,  · Link to download flutter: Another link to download Flutter: My Old channel.

NPointer Gesture- and voice-based computer control. An application for gestural and voice computer control. Hand movements are recorded by a webcam and translated into pointer movements. Gestures or voice commands a web camera for using the gesture-based computer control; a microphone for using the voice control. Representatives. skyfall - Gesture Controlled Web based Game using Tensorflow Object Detection Always Available (Body as) Input Using parts of the human body as input has the benefit of being always available as the user is not required to carry any secondary device. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back. Simple-OpenCV-Calculator. A gesture controlled calculator. Note. In the webcam feed you will see a green window (inside which you will have to do your gesture) and a .

GiMiSpace Cam Control is a free Windows hand gesture control software which allows you to control some of the mouse movements with your hands using your webcam.

Movements in front of the webcam are gonna be translated into vertical and horizontal cursor movements or zooming in effect, similar to what you get on smartphones. Desktop publishing involves using a computer and a printer, usually a laser printer to produce camera ready artwork.

There is no intermediate step - what you get out of the laser printer is the finished artwork. Check out the latest Microsoft technology, it is Gesture Controlled Computer using a Webcam. “Using a $30 dollar camera and this piece of Podtech software that’s still in development, you can play with computers just like Tom Cruise did in Minority Report, by grabbing files by the nipples and dragging them around the screen.

skyfall - Gesture Controlled Web based Game using Tensorflow Object Detection Always Available (Body as) Input Using parts of the human body as input has the benefit of being always available as the user is not required to carry any secondary device.

Gesture recognition - Wikipedia