The ISC Int'l Journal of

Information Security

ISeCure
September 2012, Volume 4, Number 1 (pp. 63–76)
[Link]

SEIMCHA: A New Semantic Image CAPTCHA Using Geometric

Transformations
Maryam Mehrnejad 1,∗ , Abbas Ghaemi Bafghi 1 , Ahad Harati 2 , and Ehsan Toreini 3
1 Information
and Communication Security Lab., Computer Department, Ferdowsi University of Mashhad (FUM), Iran
2 MachineVision Lab., Computer Department, Ferdowsi University of Mashhad (FUM), Iran
3 Computer Department, Faculty of Engineering, Islamic Azad University – Mashhad Branch, Mashhad, Iran

ARTICLE I N F O. ABSTRACT
Article history:
Received: 8 December 2011 As protection of web applications are getting more and more important every
Revised: 23 March 2012 day, CAPTCHAs are facing booming attention both by users and designers.
Accepted: 24 January 2012 Nowadays, it is well accepted that using visual concepts enhance security and
Published Online: 11 July 2012
usability of CAPTCHAs. There exist few major different ideas for designing
Keywords: image CAPTCHAs. Some methods apply a set of modifications such as rotations
SEIMCHA, Semantic Image to the original image saved in the data base, to make the CAPTCHA more
CAPTCHA, Geometric secure.
Transformation, Upright
Orientation, Tagging Image In this paper, two different approaches for designing image based CAPTCHAs
CAPTCHA, Random Guessing. are introduced. The first one—which is called Tagging image CAPTCHA—is
based on pre-tagged images, using geometric transformations to increase secu-
rity, and the second approach tries to enhance the first one by eliminating the
use of tags and relying on semantic visual concepts. In fact, recognition of up-
right orientation is used as a visual cue. The usability of the proposed approaches
is verified using human subjects. An estimation of security is also obtained by
different kinds of attacks. Further studies are done on the proposed transforma-
tions and also on the properness of each original image for each approach. Re-
sults suggest a practical Semantic Image CAPTCHA which is usable and secure
compared to its peers.
c 2012 ISC. All rights reserved.

1 Introduction Various criteria have been proposed in the litera-

ture for evaluating CAPTCHAs. We will consider the
Completely Automated Public Turing Test to Tell
following four properties (originally reported in [1])
Computer and Humans Apart (CAPTCHA) offers
in development of CAPTCHAs:
a way to make distinction between a human and an
artificial agent. Nowadays, with an increasing rate (1) Automated: Tests should be easy to be automat-
of free web services the problem of misuse through ically generated and graded by a computer.
spammers and automated soft-bots is getting worse (2) Open: The underlying database(s) and algo-
on regular basis. Therefore, it is crucial to make such rithm(s) used to generate and grade the tests
a distinction. should be public. This property is in accordance
with Kerckhoffs’s Principle, which states that a
∗ Corresponding author. system should remain secure even if everything
Email addresses: [Link]@[Link] (M. about the system is public knowledge.
Mehrnejad), ghaemib@[Link] (A. G. Bafghi), (3) Usable: Tests should be easily solved by humans
[Link]@[Link] (A. Harati),
in a reasonable amount of time. Furthermore,
etoreini@[Link] (E. Toreini).
ISSN: 2008-2045 c 2012 ISC. All rights reserved.
the effect of a user’s language, physical location,

ISeCure
64 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

dom characters -in some cases meaningful words from

dictionaries- to increase the number of outputs [6],
but in image based CAPTCHAs it is not possible to
produce a meaningful image easily. Using a limited
database is an inevitable solution for image based de-
signers. Therefore, there is a tight-coupling between
the general security of an image based CAPTCHA
and the security of image database. Note that even if
the database is not open—which breaks the rules of an
ideal CAPTCHA—the attacker can still or acquire all
or a part of the images by frequent use of CAPTCHA
and attack the CAPTCHA by using machine learning
or direct matching techniques—which are discussed in
detail later. So it is essential to develop some solutions
to improve the security of CAPTCHA independent
Figure 1. PIX CAPTCHA of its database [3]. In this way, we could prevent a
successful attack, even if we miss the security of data
base. These solutions could be:
education, and/or perceptual abilities should be
minimized. • Using unlimited databases
(4) Secure: Tests should be difficult for machines to • Updating the limited database
solve algorithmically. • Showing a transformed image instead of the orig-
inal one.
The first CAPTCHA was a text based one and
was proposed in 2000 for Yahoo in Carnegie Melon The third solution is the more common method
University [2]. After that, text based CAPTCHAs and a combination of all mentioned methods could be
began to be considered by researches widely in the applied too. In simple image CAPTCHAs no changes
last 10 years. Since designing and implementation of are made to the images and the user is asked to type
text based CAPTCHAs is simple, they are being used or select the name of the image from a list. In more
wide-spreading today. But still some people find the sophisticated CAPTCHAs, some modifications like
current text-based CAPTCHAs annoyingly difficult image rotation is applied and the user is asked to
[3]. Also there are different ways to attack a text based determine these changed images. Examples of such
CAPTCHA based on Optical Characters Recognition CAPTCHAs will be introduced in next section.
(OCR) algorithms. Chandaval et al has developed a There are different kinds of image based CAPTCHAs.
framework to attack text based CAPTCHAs [4]. They Some uses tagging or labeling which is assigning one
discussed various ways of CAPTCHA breaking using or some words to some objects in an image. Auto-
bots and proposed a framework for examining strength mated meaningful tagging by machine is a challenge
of these CAPTCHAs. in this area. Content Based Image Retrieval (CBIR)
Different methods are proposed to replace text- and Knowledge Based Image Retrieval (KBIR) al-
based CAPTCHAs including Image CAPTCHAs, gorithms can be used to obtain meaningful tags for
Video CAPTCHAs and Audio CAPTCHAs [1, 5–9]. images automatically, though it is very hard [11]. One
Also some combinations of these methods are being of the reasons why machines are weaker in tagging
used. Recognizing these media has more difficulty for than humans is the fact that humans use the back-
computers compared with Text based ones. Image ground of image to tell the tagging, which machines
CAPTCHAs are facing booming attention both by are incapable of.
users and designers due to more security and usability. In this paper, first we introduce a new tagging
Therefore they are used as a good alternative for text image CAPTCHA using geometric transformations as
based CAPTCHAs. The first idea using image to tell a more complex method. Indeed, we consider a set of
humans and machine apart is used in ESP-PIX [10]. 3D shapes such as sphere, cone and other shapes and
In this CAPTCHA which uses a limited database of wrap the original image on to one of them. Then a 2D
tagged images, some photos of a similar topic are projection from a random viewpoint gives the final
chosen and the user should guess the topic and select image and helps to generate many new various images
it from a given list. Figure 1 shows an example of this of each original image. The user should recognize the
CAPTCHA. transformed picture and finds an appropriate tag from
Text based CAPTCHA designers use a random a proposed list.
generator to produce a word containing some ran-

ISeCure
 September 2012, Volume 4, Number 1 (pp. 63–76) 65

(a) Assira CAPTCHA system (b) Collage CAPTCHA (c) 2D CAPTCHAs from 3D
models

Figure 2. Example of image based CAPTCHAs

Tagging CAPTCHA systems are difficult since users We have extended the proposed Tagging CAPTCHA
require a priori knowledge of the image tags and it is based on upright orientation concept and designed a
a language-dependent method. In addition, Machine’s novel semantic image CAPTCHA named SEIMCHA
weakness in automatic tagging, not only counted as an in which the user should click on the upper area of
advantage for designers since bots cannot attack easily, the transformed image. This is the first time that
but also is a disadvantage too as a wide range of reliable geometric transformations and upright orientation
labels that are not available for most images on the concept are applied to design a CAPTCHA system.
web to create a random challenge. So as addressed in This combination leads to a more secure and more
[5] there are some common techniques used to gather usable CAPTCHA.
proper labels for images:
Section 2 introduces related works containing prior
(1) Using the label assigned to an image by a search tagging CAPTCHAs and upright orientation based
engine, CAPTCHAs. In Section 3, the proposed Tagging
(2) Using the context of the page to determine a CAPTCHA is described. Furthermore it presents all
label, applied transformations including geometric functions,
(3) Using images that were labeled when they were Also security and usability analysis on the proposed
encountered in a different task, or Tagging CAPTCHA are described in this section. Sec-
(4) Using games to extract the labels from users tion 4 presents SEIMCHA based on upright orienta-
(such as the ESP game). tion and geometric transformations and includes all
security and usability analysis. In Section 5, we make
It is obvious that there are some limitations to ob-
some comparisons between the proposed methods and
tain labels in these methods since asking user to solve
similar works. Finally, Section 6 is conclusion and
tagging CAPTCHA system. Noisy labels, unreliable
suggested future works.
and unrelated labels, misspelling, synonymous words,
linguistics problems and etc, are the main obstacles
of these methods.
2 Related Works
One way to escape tagging CAPTCHAs is to apply
Since this paper introduces two separated CAPTCHA
semantic content in images. Semantic cues could be
systems—the Tagging CAPTCHA and SEIMCHA-
identified by users instead of selecting and/or mapping
related work falls into two main groups. The former in-
tags. It is a new solution in which limited CAPTCHAs
troduces some non-semantic image based CAPTCHAs
applied it so far. Upright orientation of an image is
from different level of distortion on images and the
a semantic which is easy for human to comprehend
latter group contains all previous works have been
and hard for machine. Currently, automatic detection
done based on upright orientation.
of such concepts is possible only for a small subset
of images [12, 13].As reported in [5] 68.75% of users There are various CAPTCHAs from basic to ad-
preferred rotating images as CAPTCHA, and 31.25% vanced which uses images without any changes, few
of users preferred deciphering text. So it seems that changes and sophisticated ones. Microsoft Assira is
upright concept of an image is a potential choice to a famous example of these CAPTCHAs in which the
use as CAPTCHA. There are a few works based on user must choose cats in a 12 image set of cats and
this idea which will be explained in next section. dogs [7]. Figure 2a shows a screen shot of Assira. Col-

ISeCure
66 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

Figure 4. Example CAPTCHA based on line drawings.

Figure 3. What’s Up CAPTCHA interface [5]

drawing was automatically rendered from a 3D model
using a randomized point of view, providing for many
lage [14] is another instance which displays some ro- possible images from each model” [3].
tated images and the user has to find an object which
the algorithm requests (Figure 2b). Improved Collage We will discuss the pros and cons of each mentioned
[8] is a promoted version of Collage in which a ran- CAPTCHA and the proposed ones in Section 5.
dom number of images were chosen and edited, and
then the user assigns each photo to their names on
the other side of the page. In more advanced systems, 3 Tagging Image CAPTCHA based
more changes are made to the pictures. In [9] a 2D on Geometric Transformations
CAPTCHA is proposed using 3D models. A limited
As it was mentioned before, first, a new method is pro-
database of 3D images is applied in this CAPTCHA,
posed which doesn’t need any huge image database or
and these pictures are converted with changes such as
having a large number of saved tags for CAPTCHA. A
rotation, brightness, size and etc. to produce an un-
constant number of 30 images are used in the database.
limited number of 2D images for showing to user. The
Each image is tagged by its own name. These tags
Graphic User Interface (GUI) asks the user to decide
which contain different subjects like animals, foods,
which tag is most suitable for the picture (Figure 2c).
different scenes and etc. are selected the same as in
It is important to notice that there are several types
[9]. Furthermore, the images and tags are not ambigu-
of such image CAPTCHAs. But since in the proposed
ous for humans. The images are transformed by some
Tagging image CAPTCHA the focus is on the transfor-
geometric transformation functions which are a novel
mations, so Assira, Collage and 2D CAPTCHAs from
approach to create a large search space from a finite
3D models are appropriate nominations for making
image database. This section explains whole steps for
comparisons.
developing, testing and evaluating the proposed Tag-
On the other hand, based on our knowledge, there ging image based CAPTCHA.
are two main works based on upright orientation of
an image. Gossweiler et al. proposed the idea of image 3.1 Transformations
orientation as a basis for an image based CAPTCHA
[5]. They called their work “What’s UP CAPTCHA” We apply some transformation functions to modify an
and introduced it in this way: “This experiment will input image. These functions include simple rotations
present a series of images one at a time. Each image and geometric transformations. Then we convert 3D
will be rotated to a random angle. Use the provided object to 2D images by capturing from a random view-
slider to rotate the image until you believe it is in point. The algorithm below describes the approach
its natural, upright position, then press submit to go better:
to the next image. This process will continue until (1) Randomly select an input image
you have adjusted ten images.” [5]. Figure 3 shows a (2) Randomly rotate input image
screenshot of What’s UP CAPTCHA system. (3) Randomly select a 3D geometric transformation
As an extension of What’s Up CAPTCHA, Ross and transform image on it
et al. introduced a new CAPTCHA based on upright (4) Capture a 2D image from a random viewpoint
orientation of line drawing rendered from 3D mod- These transformations are implemented in Matlab
els which is called Sketcha [3]. They download their software. The steps are described in the following
models from Google3D Warehouse and render a col- sections.
lection of images from various angles. A screenshot of
Sketch is shown in Figure 4. They explain Sketcha’s
3.1.1 Rotating Input Images
response mechanism in this way: “The user’s goal is to
rotate each image until it is upright, choosing among When an image is rotated by a random degree, an
four orientations by clicking on the image. Each line extra white margin in produced in the final image as

ISeCure
 September 2012, Volume 4, Number 1 (pp. 63–76) 67

(a) Input image (b) Rotated image with (a) Input image (b) Output instances
extra margin (the square
in center is imaginary) Figure 7. Input image and final warped images

3.1.3 Rendering 2D images from 3D objects

The next step is creating a 2D image from the produced
3D object. When capturing a 2D image from the 3D
(c) Output image (d) Final output image object, we imagine the camera sight is always adjusted
by ROIrotate
to the center of 3D object and the camera is turning
Figure 5. The process of rotating input image on a fixed sphere around the 3D object. Matlab uses
2 angles to turn around a 3D object; Azimuth and
Elevation. As we examined different viewpoints for
acquiring usable 2D images, we realized that some of
output images from particular angles are not usable
for users. To prevent producing some of those unusable
pictures, Elevations is adjusted between 50 and −50
degrees. Figure 8 illustrates some examples of usable
and unusable images which are taken from different
angles.

3.1.4 Improving Usability of Geometric

Transformations
In order to improving usability of geometric transfor-
mations, in addition to adjusting camera angles, we
proposed a heuristic function called H0 which applies
a heuristic image which is a white plain image with a
Figure 6. Selected geometric functions
black mark in the center as shown below (Figure 9).
H0 image is transformed with input image concur-
can be seen in figure 5-b. To remove this extra margin
rently. If H0 final image contains some part of this
some parts of output image should be cut by selecting
black mark, the main final image is tagged as usable in
a Region Of Interest (ROI) and rotate image to the
database, otherwise it is unusable. Since the center of
wanted angle. For this purpose, ROIrotate function is
an image is more important for human to identify the
applied to the algorithm. This function is accessible
whole image, as it was predicted, H0 works. Figure 10
from the Matlab website [15]. Figure 5 shows the
shows some usable and unusable images which are
process of achieving the final rotated image step by
produced by H0. In order to a better display and un-
step.
derstanding of final H0 images, the background color
is changed to yellow and some imaginary black lines
3.1.2 Geometric Transformations are added to the H0 image.
Geometric transformations are a subset of mathemat-
ical transformations. A mathematical function trans- 3.2 Tagging CAPTCHA system
forms the pixels of an image to another position in
As discussed before, in all steps of the algorithm, some
page or space. These functions are various with sev-
random variables are applied to expand the search
eral variables. However, we only have used 6 fixed 3D
space. The original images are stored in a file. For every
of them in this paper which is shown in Figure 6.
input image, the preprocessing algorithm generates 4
We used Warp function in Matlab to transform random images for all 3D transformation. Therefore,
images on these 3D shapes. One input image and the six 3D transformation functions give us 24 final
output instances is shown in Figure 7. images for every input image and 720 final images are

ISeCure
68 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

(a) Input image

(a) Unusable output (b) Usable output

Figure 10. Usable and unusable output and heuristic images

• User Information Table: includes user information

(Age, Sex, Field and Grade) with 20 records.
(b) Usable images out of the range of [-50, 50] for
Elevation
• Picture Information Table: contains image infor-
mation (Name, Transformations and result of
applying Heuristic H0) with 720 records.
• Feedback Table: includes users feedbacks per test
(Image Name, Passed or Failed and Response
Time) that has 1200 records.
These records were analyzed by a program and
the results are presented in Section 4. However some
interesting facts about the input images were exploited
(c) Unusable images in the range of [-50, 50] for Eleva- from this data which are presented in this section.
tion

Figure 8. Usable and unusable images by setting camera

3.3 Usability of Heuristic H0
angles Selected input images, selected transformations and
the proposed heuristic have some effects on the success
rate which are discussed here:
(1) Non-centric object: H1 doesn’t have a good ef-
ficiency on the images that the main subject
is not in the center. For instance, in the Light-
house image in Figure 11a, since the main dis-
Figure 9. H0 image tinguishable concept (the tower) is in the right
side of the image, applying H0 doesn’t make it
created for 30 input images. Please consider that, these too more usable.
720 images are only a possible subset of all images can (2) Multi concept images: images including more
be produced by this algorithm. than one subject- causes users to select wrong
tag for the final image. As it can be seen in
In order to evaluating the proposed methods, we
Figure 11b, there are some Fish (the secondary
tested a practical CAPTCHA system in this phase.
object) around the Dolphin (the main subject).
The GUI is implemented in [Link] and it displays an
After applying transformations, users select the
image per challenge to the user with a menu contains
second subject in some cases.
30 labels. The user selects the label relating to the
(3) Repetitive images: when the main subject is
concept of the image. We asked 20 users to response to
copied several times in an image, users can re-
Tagging CAPTCHA including 10 male and 10 female,
sponse easier to it. For example in Figure 11c, 4
18 to 30 years old, which were undergraduate and
fishes are in one image. This image is one of the
graduate students. The proposed Tagging CAPTCHA
most successful images in its response rate.
was new to all users. 30 images in 2 rounds were
displayed to each user and feedbacks were logged into Considering these tips in designing heuristic func-
an Access Database containing 3 tables: tions and in selecting input images makes the final

ISeCure
 September 2012, Volume 4, Number 1 (pp. 63–76) 69

Table 1. Success rate of 3D objects Table 2. Usability metrics

Number 1 2 3 4 5 6 Round Success rate Average

no. Without H0 With H0 response time
3D object
1 82.92% 76.83% 8.99 seconds
Without 77% 85% 76% 80% 82% 78%
2 91.06% 84.33% 6.1 seconds
H0
With H0 81% 88% 87% 86% 91% 81%
Table 3. Needed operations for direct matching attack

CAPTCHA more usable. Variable name Number of modes

Input Images 30
3.4 Response time and Success rate Rotation in step 2 of 360 degrees of freedom
algorithm
Usability of the system means how it is easy for human
to response. So they are named human metrics too. Geometric transformations 6 — at least
Two main factors are considered as usability metrics Camera viewpoint 100 × 360 × 360 degrees of
in CAPTCHAs; Response time and Success rate. First freedom
One of the degrees is set to
we present the success rate of each 3D objects in
[−50, 50] for more usable
Table 1. As it can be seen cone and sphere have the images
best and the worst response rate with H0, respectively.
All possible final images 840 milliards
In addition, generally there is a rise of 7% in usability without H0
after applying H0.
All possible final images 630 milliards
Table 2 shows the information about usability met- after applying H0
rics for the proposed Tagging image CAPTCHA in 2 Needed operations for log2 (1200 × 900) = 20.043
different rounds. Since users are more familiar with comparison 2 images
the challenges and images in round 2 the results is All needed operations 16.8 × 1012
better than round 1. And it is predictable it would be without heuristics
better in next rounds too. Users can pass the proposed All needed operations after 12.6 × 1012
Tagging CAPTCHA after one training round in 6 sec- applying heuristics
onds and with the success rate of 91% which generally
are good results in usability metrics for CAPTCHAs. 1.66 × e−18 .
On the other hand, these transformations are interest-
ing for users too and they interact with the challenge If an attacker uses the CAPTCHA too many times
as a game we are solving that. In conclusion, geomet- or (s)he steals all or some parts of input images from
ric transformations could be considered as potential database, (s)he can ask several users to exploit cor-
options for using in image based CAPTCHAs. rect tags by paying them a reasonable cost or credit—
Mechanical Turk Attack. Saving main images in DB
instead of transformed ones reduces this attack con-
3.5 Security Analysis
siderably. Now imagine the attacker has gained some
Security metrics are about the security of the or all parts of the DB in some way. S(he) can con-
CAPTCHA systems. Since they measure the strength struct a new DB as a lookup table and save all possible
of the system against the machine bots, they are called output images after applying transformations. S(he)
machine metrics too. Security metrics are divided into should compare the displayed image with all images
three main types; Random guessing, Direct matching saved already. The minimum required operation for
and Learning attacks. We present the first and second compacting two images is equal to log2 (image pix-
attacks here and the third one will be discussed later. els). The output images in the proposed CAPTCHA
system are 900*1200 pixels and based on the statis-
If an attacker selects one of the tags in the menu ran-
tics, about 25% of images are removed after applying
domly, and (s)he responses to the challenge correctly,
heuristic. Table 3 calculates all needed operations for
(s)he performs a successful random guess attack. We
direct matching attack.
should calculate the probability of this selection as
the probability of random guessing. Since there are 30 Note that geometric transformations have much
tags in the proposed GUI, this probability is equal to variety which is limited in this work. As it can be seen
3.3%. If the challenge repeats 2 times for the user, it in the table, although many variables are fixed for the
decreases near to 0.1% and if the GUI presents many practical tests, there is a huge search space for the
images for example 12 ones, this could plummet to proposed Tagging CAPTCHA which can be used in

ISeCure
70 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

(a) Main subject on the side (b) Multiple concept (c) Repetitive image

Figure 11. Some example of input images

other image CAPTCHAs too.

4 SEIMCHA
From previous section, we found geometric transfor-
mations as a potential solution to use in a more prac-
tical CAPTCHA which is more usable and secure. It (a) Input image (b) Key image
is undeniable that increasing round number to im-
prove security decreases usability because of longer
response time. Furthermore, the proposed Tagging
CAPTCHA suffers from the general problems of the
other tagging CAPTCHAs. Today users look forward
to faster response mechanism to pass CAPTCHAs.
Based on the previous works on upright orientation
CAPTCHAs, we found this concept appropriate for (c) Examples of main output images and key output images
designing more secure and usable CAPTCHAs. In
this section we present SEIMCHA as a new semantic Figure 12. main and key images
image CAPTCHA which is a combination of upright
orientation concept and geometric transformations.

4.1 Identifying Upright Oreintation

As the first challenge in using upright orientation in
a CAPTCHA, we face the problem of producing the
correct answer for the test. It means that the server—
which is sending the tests, should know the answer.
Considering the issue that we do not save the upright
orientation as a tag in DB, the server should produce
it dynamically. One of the most important advantages
of using upright orientation instead of labeling is that Figure 13. SEIMCHA interface
there is no need to keep something as key in database.
In fact, we can design an algorithm to produce the 4.2 SEIMCHA System
key on the fly. We suggest a key image transforming
We implemented a beta version of SEIMCHA in
exactly like the input image which is divided into
[Link]. The GUI presents a series of images to users
three parts and the corresponding top part of it is
in several rounds and asks them to click on the up-
considered as the answer area which could be clicked
right orientation. The clicked point is returned to the
by user as correct answer (Figure 12b). Indeed, Users
server and is checked in the corresponding key image.
should click on the logical upright orientation of an
Then, the user will be announced if s(he) passes or
image as right answer which is a specific area for the
fails the challenge (Figure 13).
server. Please note to the fleshes in Figure 12c showing
the area could be clicked by user. Again we asked 20 users to take part in SEIMCHA,
10 male and 10 female, 22 to 30 years old. They study
as undergraduate, graduate and Ph.D. students in en-

ISeCure
 September 2012, Volume 4, Number 1 (pp. 63–76) 71

Figure 14. H1 image (a) Input image (b) Key image

gineering faculty of Ferdowsi university of Mashhad 1 .

SEIMCHA was new to all users and 60 images were
displayed to each user and all feedbacks were logged
into an Access database containing 3 tables:
(c) Usable image (d) Unusable image
• User Information Table: includes user information
Figure 15. Example of applying H2
with 20 records.
• Picture Information Table: Contains information
about images (Image name, Transformations and
SEIMCHA Mode: H1 and H2 H2 H1 None
H2 and H3 tag—which are two heuristic functions) 27
with 720 records. 25
• SEIMCHA table: includes users feedbacks (Image 23
21
name, Passed or failed and Response Time) with
19
1200 records. Input Image Number 17
15
13
4.3 Improving Usability of SEIMCHA 11
9
Adding upright orientation concept changes the style 7
of response mechanism and needs new solutions to 5
raise usability. We applied two new heuristics to im- 3
1
prove usability of SEIMCHA. The first one focuses on 0% 20% 40% 60% 80%
a specific part of input image and the second one uses Failure Rate
the visible correct area in the final image.
The former which is called H1 is based on H0 using Figure 16. Difficulty rate of input images
a plain white image with a mark in the middle of top
part of it (Figure 14). Note that the lines are imaginary 4.4 Usability Analysis
and do not exist in the main image. Since the top SEIMCHA has a lot of aspects to be investigated as
part of an image is more important for identifying its a variable affecting usability metrics. In this section,
upright, the mark is transferred to the top part of it. we report all analysis we performed on SEIMCHA
Second heuristic, H2, uses the key image. After feedback database.
applying transformations, when the final key image Input Images and 3D Objects analyzed failure and
is generated, a program calculates the percentage of success rate of input images to find which sort of
black part of it as visible correct answer. If the correct them are more appropriate for SEIMCHA. Figure 16
answer area was less than 20% of color parts of whole shows difficulty rate of these 30 images in 4 modes
image -not white margin parts- the final main image of SEIMCHA; without heuristics, with H1, with H2
is marked as unusable in database. Figure 15 shows and with 2 heuristics. There are four input images
some examples of H2 output. that their failure rate after applying H1 and H2 is still
Consider that the image in Figure 15d is not unus- more than 50% (Figure 17). It is interesting to note
able by itself. But since the visible key region is not that these images have multiple upright orientations.
enough to click, the user cannot response to the chal- According to our previous works, users can recognize
lenges presenting such sort of images. Foe fixing this these images well [16] but they cannot identify the
problem, the answer area could be defined like a fuzzy upright orientation of them.
variable which can be developed as a good point for Also we analyzed six 3D objects we applied in SEIM-
future works. CHA. Table 4 shows usability of each 3D object. The
sphere has the most failure rate before applying heuris-
tics, but it is changed to the best shape after that.
1 [Link] Other 3D objects have almost the same failure rate.

ISeCure
72 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

SEIMCHA Mode: None U1 U2 U1 and U2

0.9
0.85
(a) (b) (c) (d) 0.8

Success Rate
0.75
Figure 17. Example of unusable images 0.7
0.65
Table 4. Usability of 3D objects (Failure Rate) 0.6
0.55
Number 1 2 3 4 5 6 0.5
1 2 3 4 5 6 7 8 9 10 11 12
3D object Expiration Time

Without 40% 32% 53% 35% 39% 38%

Figure 19. Relationship between success rate and expiration
heuristics
time
With H1 32% 26% 17% 26% 26% 29%
and H2 and add some new images with less hardness rate.
On the other hand, since simple response mechanism
30 of SEIMCHA—single clicking—response time is too
General Failure Rate of CAPTCHA

short. Table 6 shows response time average in 4 modes

25 100%
of SEIMCHA.
20 90%
83% Remained Image Rate in DB Average response time is 4.03s for all challenges and
15 70% 3.81 s for success challenges. We defined an expiration
10
53% time and used the time to find whether challenges
27% with more response time improve success rate or not.
5 Figure 19 shows success rate depend on the expiration
0 1% time.
۶٣ ۵۵ ۴۵ ٣۵ ٢۵ ١۵ ۵
Maximum Failure Rate of Images It can be concluded from Figure 19 that users usu-
ally are successful in challenges in shorter time and
challenges with long response time are difficult for
Figure 18. Effect of removing hard images on SEIMCHA
users and will be failed by them.
failure rate and remained image rate

Success Rate and Response Time Table 5 shows 4.5 Security Analysis
SEIMCHA results for 4 modes. These results are before
filtering input image database of difficult images for We discussed 3 types of attacks before. The probability
human. of random guessing—as the first attack—is reported
in Table 7 for SEIMCHA. A final image contains
As we discussed in the previous section, some images some white margins between 30% and 80%, and in
are not appropriate to be applied in SEIMCHA. We average about 50%. An attacker could perform some
can remove hard images from database. Figure 18 preprocessing to click on a colorful point of image. As
shows the effect of removing hard images on general discussed before, we adjust the minimum margin of
failure rate and remained images rate in database correct answer to 20% of whole image (colorful parts).
when SEIMCHA works with two heuristics. Random guessing success rate could be improved by
As it is shown in Figure 18, we can remove the images decreasing the minimum margin of correct answer
with 25% failure rate and also more. General failure area.
rate of SEIMCHA will decrease to 10% and it means This probability is for one image per test. A SEIM-
that success rate increases to almost 90%. However in
this case, we have to remove 55% of all input images Table 6. Response time

Table 5. Failure rate of SEIMCHA before filtering input images SEIMCHA Without H1 H2 H1 and
mode heuristics H2
SEIMCHA Without H1 H2 H1 and
All 4.18 s 4.11 s 4.02 s 4.03 s
mode heuristics H2
challenges
Failure rate 39% 34% 26% 25%
Success 3.87 s 3.87 s 3.79 s 3.81 s
Success rate 61% 66% 74% 75% challenges

ISeCure
 September 2012, Volume 4, Number 1 (pp. 63–76) 73

Table 7. Random guess attack probability further experiments to compare these works.

SEIMCHA mode Without H1 H2 H1 and 5.1 Usability Metrics and Security Metrics
heuristic H2
Percentage of answer 12.32% 13.5% 17.4% 17.5% Unfortunately, there are no response time and success
area without any rate for Collage and 2D CAPTCHAs from 3D models
preprocessing [9, 14]. Assira has 83.4% success rate and 15 seconds
Percentage of answer 24.62% 27.11% 34.79% 35.05% for selecting 6 images out of 12 in one round. And if
area with the challenge repeats 3 times these numbers go up to
preprocessing 99.96% and 45 seconds respectively [7]. Again there is
no response time for What’s Up but it is 35 seconds for
CHA system which displayed 8 images would achieve 10 images in Sketcha. Success rate is 84% for 3 images
a guess success rate of less than 0.7 × 10−6 %. in What’s UP and 88% in Sketcha [3, 5]. Finally in
this work, users are successful is just over 91% and
The second attack is direct matching. As we cal-
in about 6 seconds in Tagging CAPTCHA and about
culated in Tagging image CAPTCHA, an attacker
90% and in 4 seconds for SEIMCHA.
should perform 16.8 × 1012 operations on it. When
heuristics are applied in SEIMCHA, about 20% of Also random guessing attack in Assira is 0.39% for
images will be removed. So, the needed operations 8 and 0.024% for 12 images. This is about 16% for
are 13.44 × 1012 which is a considerable time and is a Collage while it is about 3.3% for 2D CAPTCHAs
great deal for the attacker. from 3D models and the proposed Tagging CAPTCHA
for 1 image. It is 25% for 1 image in Sketcha and
And finally, the third kind of attacks are machine
it decreases to 0.001% for 8 images. The probability
learning based ones which use some learning methods
of random guessing is 4.44% for 1 image in What’s
to act like a human to answer the challenge in the
UP and again it reduces to 0.009% for 3 images. And
CAPTCHA. Fortunately, using geometric transfor-
finally, it is 17.5% for the proposed SEIMCHA system
mation functions distorts topology of point features
based on the statistics in the log database.
and the shapes in the image which are two common
ways to learning an image [17, 18]. In addition, adding One of the most important things makes a
upright orientation concept makes SEIMCHA more CAPTCHA system interesting to users is its response
difficult for machine since it should identify top part mechanism. Assira and Collage are interesting since
of an image more than recognizing the content of it in their GUI provide users a single click mechanism. But
the form of a label. However, it is important to notice 2D CAPTCHAs from 3D models and the proposed
that we didn’t design any special machine learning Tagging CAPTCHA are weak in this property because
system to pass SEIMCHA and we just discussed the they ask user to search in a list for the proper label.
strength of the proposed approaches in theory. Trying User’s response type of What’s Up is image rotation
to design such attack systems would be interesting as using slider or mouse movement or up-down control.
another point for future works. Sketcha requires the user to rotate each image in a set
of drawing until everyone is upright, by clicking to
turn them 90 degrees at a time. These tasks need more
5 Comparison time than a single click on the image which is provided
in SEIMCHA. Table 8 summarizes these comparisons.
As it mentioned in Section 2, there are some
CAPTCHA systems using other transformations. 2
more similar works are Collage [8] and 2D CAPTCHAs 5.2 Further Experiments
from 3D models [9]. The former only uses rotations, It is possible to automatically identify the images by
but the latter changed the color, light and some using reverse indexing image search engines like Tin-
other distortions on images. In addition, we intro- [Link]. If we want to categorize it as an attack, it is
duced Assira as one of the most famous image based a kind of direct matching. TinEye finds exact and al-
CAPTCHA systems which ask user to recognize cats tered copies of the images that you submit, including
and dogs [7]. Also What’s Up [5] and Sketcha [3] are those that have been cropped, color adjusted, resized,
two main CAPTCHAs asking user to identify upright heavily edited or slightly rotated [19]. Indeed, TinEye
orientation of images. What’s Up displayed input produces a digital signature of finger print for submit-
images with random rotations and Sketcha shows line ted image and compare it with all finger prints of saved
drawing images rendered from 3D models. In this images. We examined the strength of TinEye to find
section, we aim to discuss usability metrics and secu- rotated images. In this experiment, a rotated image
rity metrics for these CAPTCHAs and the proposed in the range of [-20, 20] was submitted to [Link]
CAPTCHAs. Then, we present some interesting

ISeCure
74 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

Table 8. Usability and Security Metrics Comparison

CAPTCHAResponse time Success rate Random guessing Response mechanism

Assira [7] 15s / 1 round 83.4% / 1 round 0.39% / 8 images Single click / 1 image
45s / 3 rounds 99.96% / 3 rounds 0.024% / 12 images Multiple click / 1 round
Collage [8] Not reported Not reported 16% / 6 images Multiple click / 1 object
2D from Not reported Not reported 3.3% / 1 image Selecting label from list
3D [9]
Tagging 6s / 1 image 91% 3.3% / 1 image Selecting label from list
CAPTCHA
What’s Up Not reported 84% / 3 images 4.44% / 1 image Slider / Moving mouse /
[5] 0.009% / 3 images Up-down control
Sketcha [3] 35s / 10 images 88% 25% / 1 image Multiple click / 1 image
0.001% / 8 images
SEIMCHA 4s / 1 image 90% 17.5% / 1 image Single click / 1 image

Figure 20. Identification rate by [Link] based on rotation Figure 21. Identification rate by [Link] based on bright-
rate ness rate

step by step. TineEye can find rotations between -15

to 15 degrees which can be seen in Figure 20. 6 Conclusion and Future Work
However it is easy for a program to rotate an image A common approach to improve the security of an
in different degrees, and then submit to TinEye and image based CAPTCHA is to display a transformed
find the main image. version of an image to user instead of the main image
Again the above experiment was repeated for bright- saved in database. Geometric transformations were
ness. We changed the image light from -100% to 100% presented as a new successful solution in this paper
from dark to light. As it can be resulted from Fig- since however many variables were fixed for our ex-
ure 21, [Link] can recognize the distorted image periments, the final search space is still too large to
in a high rate. It is noticeable that images out of [-60%, traverse by an attacker in a direct matching attack.
60%] are not recognizable by human. Then [Link] Usability of the Tagging image CAPTCHA proposed
can find them better than human in some cases. based on geometric transformations is better than
the similar works and the transformations increase
The experiment has been done with the geometric security too. As it can be seen in table 8 all metrics
transformations proposed in this work. Fortunately, except the probability of random guessing is improved
[Link] couldn’t find any version of the input image in Tagging image CAPTCHA.
which is a promising result for this work.
Furthermore, we presented a new semantic image
Today, designers consider such attacks to evaluate a based CAPTCHA named SEIMCH which is a none-
CAPTCHA system [2]. It can be concluded from above tagging CAPTCHA using upright orientation and
experiments that Collage, 2D CAPTCHAs from 3D geometric transformations. Applying these two, pro-
models, Assira and What’s Up are weak to [Link]. vides a more usable and secure practical CAPTCHA
whereas, Sketcha and SEIMCHA are robust to this which eliminates problems of the proposed Tagging
search engine. image CAPTCHA. SEIMCHA has a simple response
mechanism -single clicking- which makes it faster than
similar works. Finally, by selecting an appropriate set

ISeCure
 September 2012, Volume 4, Number 1 (pp. 63–76) 75

of input images, SEIMCHA has an excellent response Persian/arabic baffletext captcha. Journal of
time and success rate which is about 4 second per universal computer science, 12(12):1783–1796,
each image and about 90%, respectively. This gives 2006.
the confidence to extend the proposed approaches. [7] J. Elson, J.R. Douceur, J. Howell, and J. Saul.
Asirra: a captcha that exploits interest-aligned
As future works, we suggest below:
manual image categorization. CCS, 7:366–374,
• Using more mathematical functions for transform- 2007.
ing images and exploiting the best set of them. [8] R. Soni and D. Tiwari. Improved captcha method.
• Using a fuzzy method to identify upright orienta- International Journal of Computer Applications
tion instead of a 0 and 1 mechanism. IJCA, 1(25):107–109, 2010.
• Designing a Multiple SEIMCHA which shows [9] M.E. Hoque, D.J. Russomanno, and M. Yeasin.
several images in the GUI instead of one image to 2d captchas from 3d models. In SoutheastCon,
reduce the probability of random guessing attack. 2006. Proceedings of the IEEE, pages 165–170.
• Applying a mechanism to update the database IEEE, 2005.
of input images continuously to improve security [10] The CAPTCHA Project. [Link]
and also to replace images with more usable im- net/captchas/pix/.
ages. [11] T. Pavlidis. Why meaningful automatic tagging
• Designing an ‘almost right’ response mechanism of images is very hard. In Multimedia and Expo,
instead of complete right answer. It would im- 2009. ICME 2009. IEEE International Confer-
prove security more than before [7]. Imagine ence on, pages 1432–1435. IEEE, 2009.
SEIMCHA displays 3 images to users. A complete [12] L. Zhang, M. Li, and H.J. Zhang. Boosting im-
right answer means 3 correct identifications and age orientation detection with indoor vs. out-
an almost right answer means 2 correct answers door classification. In Applications of Computer
out of 3 images. Vision, 2002.(WACV 2002). Proceedings. Sixth
• Designing a particular attack on SEIMCHA that IEEE Workshop on, pages 95–99. IEEE, 2002.
uses several types of attacks. [13] S. Lyu. Automatic image orientation determina-
• And finally, studying SEIMCHA in the context of tion with natural image statistics. In Proceedings
a working website involving vast number of users. of the 13th annual ACM international conference
on Multimedia, pages 491–494. ACM, 2005.
[14] M. Shirali-Shahreza and S. Shirali-Shahreza. Ad-
References vanced collage captcha. In Information Technol-
[1] K.A. Kluever and R. Zanibbi. Balancing usability ogy: New Generations, 2008. ITNG 2008. Fifth
and security in a video CAPTCHA. In Proceed- International Conference on, pages 1234–1235.
ings of the 5th Symposium on Usable Privacy and IEEE, 2008.
Security (SOUPS), page 14. ACM, 2009. [15] Vlad Atanasiu. ROIRotate Function; A
[2] L.V. Ahn, M. Blum, N.J. Hopper, and J. Lang- Function to Fill Corners of Rotated Image.
ford. Captcha: Using hard ai problems for security. Available from [Link]
In Proceedings of the 22nd international confer- matlabcentral/fileexchange/1825, Updated
ence on Theory and applications of cryptographic 2008.
techniques, pages 294–311. Springer-Verlag, 2003. [16] M. Mehrnejad, A. Ghaemi, A. Harati, and E. Tor-
[3] S.A. Ross, J.A. Halderman, and A. Finkelstein. eini. A new image based CAPTCHA based on ge-
Sketcha: a captcha based on line drawings of 3d ometric transformations. In 8thInternational ISC
models. In Proceedings of the 19th international Conference on Information Security and Cryptol-
conference on World wide web, pages 821–830. ogy, FUM, Iran, 2011.
ACM, 2010. [17] D.G. Lowe. Object recognition from local scale-
[4] AA Chandavale, AM Sapkal, and RM Jalnekar. A invariant features. In Computer Vision, 1999. The
Framework to analyze the security of Text based Proceedings of the Seventh IEEE International
CAPTCHA. International Journal of Computer Conference on, volume 2, pages 1150–1157. Ieee,
Applications IJCA, 1(27):127–132, 2010. 1999.
[5] R. Gossweiler, M. Kamvar, and S. Baluja. What’s [18] S. Belongie and J. Malik. Matching with shape
up captcha?: a captcha based on image orien- contexts. In Content-based Access of Image and
tation. In Proceedings of the 18th international Video Libraries, 2000. Proceedings. IEEE Work-
conference on World wide web, pages 841–850. shop on, pages 20–26. IEEE, 2000.
ACM, 2009. [19] TinEye. [Link]
[6] M.H. Shirali-Shahreza and M. Shirali-Shahreza.

ISeCure
76 SEIMCHA: A New Semantic Image CAPTCHA . . . — M. Mehrnejad, A. G. Bafghi, A. Harati, E. Toreini

Maryam Mehrnejad was born in 1986 in Ahad Harati was born in 1978. He re-
Sabzevar, Iran. She received her BS and ceived his BS in Computer Engineering from
MS in Computer Engineering from Ferdowsi Amirkabir University of Technology (2000)
University of Mashhad (FUM) in 2009 and and his MS in Artificial Intelligence and
2011, respectively. She was a member of Robotics from University of Tehran (2002).
Security Information and Communication In 2003, he joined Autonomous System Lab-
Lab in FUM and also a member of FUM oratory at Swiss Federal Institute of Tech-
CERT Lab during her studies. Her main nology in Lausanne (EPFL) and two years
research interests are Security Engineering, HCI-Sec (Human later along with other colleagues moved to Zurich. He got his
and Computer Interaction and Security) and Applied Soft PhD in Robotics in 2008 from ETHZ (Swiss Federal Institute
Computing. of Technology in Zurich). Later he moved back to Mashhad and
joined Ferdowsi University of Mashhad, where he is currently
Abbas Ghaemi Bafghi was born in April an Assistant Professor. His main research interests include
1973 in Bojnord, Iran. He received his BS Range Data Processing and Multiresolution Analysis, Image
degree in Applied Mathematics in Computer Processing and Vision, Simultaneous Localization and Map-
from Ferdowsi University of Mashhad, Iran ping, Human Machine Interaction, and Multiagent Learning.
in 1995, and MS degree in Computer engi-
neering from Amirkabir (Tehran Polytech- Ehsan Toreini was born in September 1984
nique) University of Technology, Iran in 1997. in Ghazvin. He is MS graduate of Islamic
He received his PhD degree in Computer Azad University, Mashhad Branch in 2010
engineering from Amirkabir (Tehran Polytechnique) Univer- and BS graduate of Ferdowsi University of
sity of Technology, Iran in 2004. He is member of Computer Mashhad in 2007. He is now a lecturer in
Society of Iran (CSI) and Iranian Society of Cryptology (ISC). Islamic Azad University, Mashhad Branch
He is an assistant professor in Department of Computer Engi- and member of Young Researchers’ Club.
neering, Ferdowsi University of Mashhad, Iran. His research His main fields of study are Data Mining,
interests are in cryptology and security and he has published Machine Learning and Computational Intelligence.
more than 50 conference and journal papers.

ISeCure