Abstract proposed system describes a new method for



An Individual
Human Computer Interface system using eye movement is  introduced. Traditionally, human computer interface uses mouse or keyboard
as  an 
input  device. The proposed system
presents hands free interface between computer and user .The main objective is
to control the Mouse & Keypad using Eye ball. It also verify the user’s
authentication using Face Recognition. For Face recognition, Violo Jones
Algorithm is used. Camera is connected with the system & mat lab is used
for User Authentication. After successful authentication, camera is continued
to scan User’s Eye ball movement. During this state of action, Our Physical
Keypad & Mouse are freezed in order to stop user’s Key inputs. On-screen
Keyboard & Mouse control is initiated so as to control those through Eye
Ball movements. Mat lab software plays a vital role in controlling the on-screen
Keyboard & Mouse. We will be using Java for freezing the physical keypad
& mouse functionalities. Camera scans the Eye ball of the authenticated
user and control of the mouse is achieved through the Eye ball movement.
Alphabets are selected by Eye ball clicking for effective communication. The
Physical keyboard control is released by Control, Alt & Delete keys.  Mat lab is used for Face recognition &
Eye ball control. On-screen Keyboard & Mouse are initiated & freezing
of Physical Mouse & Keypad is achieved by Java software.

We Will Write a Custom Essay Specifically
For You For Only $13.90/page!

order now


Keywords—On-screen keyboard, Face recognition and Eye detection,
Web Camera, MAT lab, Computer, Viola Jones Algorithm,



1.      Introduction


In today’s world
technology gets upgraded to the newest level, majority computers rely on mouse
and keyboard as the major input devices which could not be used by handicapped
people. The proposed system describes a new method for the handicapped people
to communicate using computers with the help of eyes only. Most of the devices
such as computers and laptop prefer touch screen technology, but still the
preferred technology is not cheap enough to be used on desktop systems. The
main aim is to develop an interactive virtual human computer interface.

In our system,
we prefer the usage of Matlab to detect the web camera which is used for taking
images continuously to focus the eye pupil. With the help of various image processing
techniques, face recognition and eye tracking are done. For Face recognition,
Violo Jones Algorithm is used.




Now a days, people use computer by their
hands and  touch pad. Traditionally,
human computer  interface uses mouse
and  keyboard as an input device. An  idea to control computer  mouse cursor movement with human eyes is
introduced1. Blink actions are introduced  to replace the mouse clicks2. Generally for
opening a file, one must click on the file by using physical mouse or touch
pad. Instead a new system is introduced to replace the physical mouse. One can
open a file using the eye movement and blink actions. Both  the left and right click is done by blinking
the left eye and right eye.









The main
objective is to control the Mouse & Keyboard using Eye ball for handicapped
people. On-screen keyboard is displayed on the desktop to replace the physical
keyboard. Camera is mounted on the top of the desktop and the user  image infront of  the computer is captured. The image is
compared and verified with the database  for
user’s authentication.    After    successful 

keyboard is displayed on the desktop automatically. Once the On-screen keyboard
is displayed, the physical keyboard and mouse is freezed. The Physical keyboard
and mouse freeze is released by pressing Control, Alt & Delete keys. Camera
is continued to scan for face recognition using Violo Jones Algorithm. Then the
eye is detected successfully to control the mouse and on-screen keyboard. Based
on the blink action, the letter is typed and displayed for
effective communication.


eye structure


The human eye is
an important organ that senses light. The important parts of human eye related
to eye tracking are described . The transparent coat in front of eyeball is
cornea. The muscle that controls the size of pupil is called iris, which is
like the aperture in a camera to let light goes inside. The iris has colour and
is different from person to person, thus can be used in biometrics. The tough
outer surface of the eyeball is sclera and appears white in the eye image. The
boundary between the sclera and the iris is called limbus. The captured eye
image by digital camera is shown in Figure below.


















specialized video camera (Logitech C170) is mounted above the desktop to
observe the user’s eyes who sits in front of the desktop. The camera captures
the video image of the eye and determines where the user is looking on the
screen. No attached is necessary to the user’s head or body. To “select” any
letter on the On-screen keyboard , the user looks at the letter for a specified
period of time and to “press” any letter, the user just needs to blink the eye.
In this system, no calibration procedure is  required. For this system, input is only eye.
External hardware is not attached or required.                  

keyboard display


Proposed Algorithm


The detailed processing steps
is presented below:


Logitech C170 camera is fixed on the top of the

The camera takes the image of the user who sits
in front of the computer.

The image is compared and verified with the
database for user’s authentication.

After successful user’s authentication,
on-screen keyboard is displayed automatically on the desktop.

Once on-screen keyboard is displayed, the
physical keyboard and mouse are freezed by using java code.

Freezing can be released by pressing control,
alt and delete keys on same time.

Once on-screen keyboard is displayed, Camera
starts scanning the user eye by taking video.

After receiving these streaming videos from the
camera, it then breaks into frames.

9.       After receiving frames, it checks for lighting
conditions because the camera requires sufficient lights from external sources
otherwise  an error message will display
on the screen.

10.    The captured frames that are already in RGB mode are
converted into Black ‘n’ White in order find the edge movement.

11.    Images (frames) from the input source focusing the eye
are analyzed for Iris detection (center of eye) using Viola Jones Algorithm.

12.    After that, a mid point is calculated by taking the
mean of left and right eye centre point.

13.    Then, the mouse will move from one point to another on
the screen and user will perform clicking action by blinking their eyes.

Based on the blink action, the letter
is typed and displayed for effective communication.


Jones Algorithm

The viola jones
algorithm is object detection framework widely used for face recognition. It
was mainly invented for the problem of face detection use to
detect  the faces in a image. An unique feature of
the algorithm is such that it has very high accuracy
& very low false-positive
rate. The algorithm works as follows:


1.Haar Feature Selection

A few properties common to human faces:

The eye region is
darker than the upper-cheeks.

The nose bridge region
is brighter than the eyes.

Composition of properties forming matchable
facial features:

Eyes, mouth, bridge of
nose –   Location and size

Oriented gradients of
pixel intensities – Value

2.Creation of Integral

Based on above four characteristics an integral image(image of a
face) which evaluates rectangular features in constant time.

3.Adaboost Algorithm

technique is preferred for detecting face and non-face.

4.Cascade classifier

low rate values are neglected and positive 
sub-window values are considered.



This paper is
fully focused on the development of 
hands-free PC control- Eyeball movement based cursor and keyboard
control. This system does not require any wearable attachment is the most
unique aspect. The mouse pointer is operated using the user eye movement. This system
has been implemented in Windows 8.1 with 4GB RAM in Matlab environment using
Matlab Image Acquisition Processing Toolbox.

Image Processing
Toolbox is sufficient for the sensitivity of the system and to work in real
time as how    normal    mouse   perform     movement task.

keyboard is introduced to perform all keyboard  
actions   for   effective  

Our motive is to
create this technology in the  lowest
cost possible way and also to create it under a standardized operating system
in more user friendly manner.

10.  References

1 Craig
Hennessey, Jacob Fiset “Long Range Eye Tracking: Bringing Eye Tracking into the
Living Room”, IEEE, 2016

2Shyam Narayan
Patel, V. Prakash ” Autonomous Camera based Eye Controlled Wheelchair system
using Raspberry-Pi”, IEEE
Sponsored 2nd International Conference on Innovations in Information Embedded
and Communication Systems, 13 August 2015

3Shrunkhala Satish Wankhede,
Mrs.S.A.Chhabria,Dr.R.V.Dharaska ” Controlling Mouse Cursor Using Eye
Movement”, Special Issue for National Conference On Recent Advances in
Technology and Management for Integrated Growth 2013 (RATMIG 2013)

4Akhil Gupta, Akash Rathi, Dr. Y. Radhika, “HANDS-
Journal of Scientific and Research Publications, Volume 2, Issue 4, April 2012


5Sidra Naveed,
Bushra Sikander, and Malik Sikander Hayat Khiyal “Eye Tracking System with
Blink Detection”,IEEE,2012

Jixu Chen, Yan Tong ,Wayne Grayy ,Qiang Jiz “A Robust 3D Eye Gaze Tracking
System”,IEEE ,2011


Ionita M., Corcoran,P. , “Statistical models of appearance for eye tracking and
eye-blink detection and measurement”, IEEE Transactions, August 2010

8Ioana Bacivarov,
Mircea Ionita, Peter Corcoran, “Statistical models of appearance for eye
tracking and eye blink detection and measurement”, IEEE transactions on
consumer  electronics, Vol.54 , No.3, pg.
1312?1320 August 2009.