1 Human Computation Play a Game to Develop an Ontology Peyman Nasirifard p+e+y+m+a+b-b+n dot deri.org.

Presentation on theme: "1 Human Computation Play a Game to Develop an Ontology Peyman Nasirifard p+e+y+m+a+b-b+n dot deri.org."— Presentation transcript:

1 1 Human Computation Play a Game to Develop an Ontology Peyman Nasirifard p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org

2 2 Agenda Introduction CAPTCHA CAPTCHA Games with a purpose ESP game ESP game Peekaboom Peekaboom Verbosity Verbosity Possible game for developing simple ontologies Play a game Conclusion

3 3 Introduction Human-based computation is a technique when a computational process performs its function via outsourcing certain steps to humans.

4 4 Back to History Yahoo! and Gmail are not interested to enable a bot to create thousands accounts per day for sending spam They use CAPTCHA to prevent it They use CAPTCHA to prevent it plus

5 5 CAPTCHA Stands for “Completely Automated Public Turing test to tell Computers and Humans Apart” Luis von Ahn et al. coined the term in 2000 A Program that can tell whether a user is a human or a computer Many different techniques

6 6 Some Examples

7 7 Dog or Cat? Human : mmm… dog Computer : mmmmmmmmmmmmmmm…

8 8 Human Computation If we use people to break CAPTCHA, we are doing human computation In some countries, some companies hire people to break CAPTCHA and send spam In some countries, some companies hire people to break CAPTCHA and send spam Some companies cleverly use humans to break CAPTCHA and send spam Some companies cleverly use humans to break CAPTCHA and send spam How? How?

9 9 Clever spammers

10 10 Clever Spammers Type the word in the box if you want to see the next picture Free Nude Photos

11 11 Really?! Jan 2004: world without spam by 2006! Huge amount of investment Huge amount of investment Bill Gates receives 4 million spams per day Bill Gates receives 4 million spams per day

12 12 Nice Quote Luis von Ahn: Instead of hiring people and pay them to solve our problems, we can design games and people will pay us to play our games and solve our large-scale problems!

13 13 The ESP game Object of the game: type the same word Only thing in common is: an image Players Do not know each other (randomly paired) Do not know each other (randomly paired) Can not communicate Can not communicateAdvantages: Two different sources labels the image Two different sources labels the image enjoyable enjoyable labels all images on Google image in a short time labels all images on Google image in a short time Help to improve English! Help to improve English! There are many people that play over 20 hours a week There are many people that play over 20 hours a week

14 14 The ESP game Player 1 Player 2 CAR GIRL TREE WOMAN CAR Agree: CAR Get points

15 15 Taboo words More difficult, but more fun More difficult, but more fun CAR WOMAN

16 16 Single version of ESP game CAR GIRL TREE WOMAN CAR The engine records everything from previous players A single player will actually play with another player, but not at the same time

17 17 Cheating and Repetition Problem: Agreement on cheating Let’s label all images with “dog” Let’s label all images with “dog” Solution: At random, system gets players test images to check whether they play honestly or not If they do not play honestly, the system will let them play, but nothing will be recorded If they do not play honestly, the system will let them play, but nothing will be recorded For certainty, only labels which at least N pairs agreed upon will be stored

18 18 The Limitations of ESP The ESP Game can label images (and consequently tell you what’s in them), but it cannot: Find the objects being labelled Find the objects being labelled Determine the way in which the object appears – does the label “car” refer to the text “car” or an actual car in the image? Determine the way in which the object appears – does the label “car” refer to the text “car” or an actual car in the image?

19 19 The place of objects in an image Such information would be extremely useful for computer vision research Such information would be extremely useful for computer vision research dog man

20 20 The Revealer clicks on parts of the image and shows them to the Guesser. The Guesser guesses: Flower Petal Butterfly Server: Correct, Butterfly

21 21 Hints The label “car” is ambiguous -- this is “car” this is also “car” The hints help distinguish the manner in which the label “car” appears: this is the object “car” this is the text “car”

22 22 Verbosity Collect common-sense facts Water quenches thirst Water quenches thirst Sky is blue Sky is blue Lions eat meat Lions eat meat We as human know hundreds of millions common sense facts Computers do not know Computers do not know If know, potentially make them more intelligent (e.g. search better)

23 23 Common sense fact samples Milk It is liquid It is white it has lactose cereal is eaten with it

24 24 Verbosity Narrator MILK is typically near cereal is a liquid Guesser

25 25 Verbosity Narrator MILK Guesser is typically near cereal is a liquid MILK

26 26 Verbosity Narrator Object Common sense facts about the object Guesser

27 27 Verbosity Narrator Object Guesser Common sense facts about the object Object

28 28 Templates ___ is a kind of ___. Allows for hierarchical categorization. ___ is used for ___. Provides information about the purpose of a word. ___ is typically near/in/on ___ (three templates). Provide spatial data. ___ is the opposite of ___ / ___ is related to ___ (two templates). Provide data about basic relations between words. ___. In the game, this is a “wildcard” that collects related words.

29 29 Symmetric vs. Asymmetric Verbosity is a asymmetric game, whereas ESP game is a symmetric game. Symmetric games: constraint is number of outputs per input Asymmetric games: constraint is number of inputs that produces the same output

30 30 Possible game to build an ontology Several game should work together Images come from ESP game Images come from ESP game Not always: only those images are selected which have one object in it Not always: only those images are selected which have one object in it i.e. car, bike, monitor, mouse, house i.e. car, bike, monitor, mouse, house These images are input to next game which tries to catch the properties of objects These images are input to next game which tries to catch the properties of objects car has colour, car has wheels, car has manufacture, car has owner, car has building year, etc. car has colour, car has wheels, car has manufacture, car has owner, car has building year, etc.

31 31 Possible game to build an ontology Cardinality will be caught by templates, as soon as we have properties. Car has four wheels Car has four wheels Car has one plaque Car has one plaque These sentences will be transferred to OWL representation using a mediator. The more pairs play the game, the more complex the ontology will be

32 32 Contact me if you are interested to work on it

33 33 This page intentionally left blank This page intentionally left blank

34 34 Guess what! It has usually four wheels It has usually one seat It is kind of vehicle It operates with human power It operates with batteries It has a break system It is a kind of chair

35 35 Answer

36 36 Conclusion Games are enjoyable! More than 9 billion Human-hours of solitaire are played each year We may cleverly using humans to solve large-scale problems by designing interesting games Many people play word-guessing games to improve their English Go and play to promote science!

37 37 References [1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf http://www.cs.cmu.edu/~biglou/Verbosity.pdf [2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf http://www.cs.cmu.edu/~biglou/Peekaboom.pdf [3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf http://www.cs.cmu.edu/~biglou/ESP.pdf [4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf [5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation http://en.wikipedia.org/wiki/Human-based_computation [6] We'll End Spam Within 2 Years, http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587 http://news.bbc.co.uk/2/hi/business/3426367.stm http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587 http://news.bbc.co.uk/2/hi/business/3426367.stm http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587 http://news.bbc.co.uk/2/hi/business/3426367.stm [7] CAPTCHA, http://en.wikipedia.org/wiki/Captcha, http://www.captcha.net http://en.wikipedia.org/wiki/Captcha http://www.captcha.net http://en.wikipedia.org/wiki/Captcha http://www.captcha.net [8] ESP game, www.espgame.org www.espgame.org [9] Peekaboom game, http://www.peekaboom.org/ http://www.peekaboom.org/ [10] Verbosity game, www.peekaboom.org/verbosity/ www.peekaboom.org/verbosity/ [11] Presentation, http://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppt http://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppt [12] Presentation, www.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.ppt www.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.ppt

38 38 Game Over Game Over p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org