THE ESP GAME, & PEEKABOOM LUIS VON AHN CARNEGIE MELLON UNIVERSITY.

Presentation on theme: "THE ESP GAME, & PEEKABOOM LUIS VON AHN CARNEGIE MELLON UNIVERSITY."— Presentation transcript:

1 THE ESP GAME, & PEEKABOOM LUIS VON AHN CARNEGIE MELLON UNIVERSITY

2 LABELING IMAGES WITH WORDS MARTHA STEWART FLOWERS SUPER EVIL STILL AN OPEN PROBLEM

3 IMAGE SEARCH ON THE WEB USES FILENAMES AND HTML TEXT

4 ACCESSIBILITY LESS THAN 10% OF THE WEB IS ACCESSIBLE TO THE VISUALLY IMPAIRED REASON:MOST IMAGES DON’T HAVE PROPER CAPTIONS

5 DESIDERATA A METHOD THAT CAN LABEL ALL IMAGES ON THE WEB FAST AND CHEAP

6 USING HUMANS THE ESP GAME COULD LABEL ALL IMAGES ON THE WEB IN 30 DAYS! THE ESP GAME CLEVERLY

7 TWO-PLAYER ONLINE GAME PARTNERS DON’T KNOW EACH OTHER AND CAN’T COMMUNICATE OBJECT OF THE GAME: TYPE THE SAME WORD THE ONLY THING IN COMMON IS AN IMAGE THE ESP GAME

8 PLAYER 1PLAYER 2 GUESSING: CARGUESSING: BOY GUESSING: CAR SUCCESS! YOU AGREE ON CAR SUCCESS! YOU AGREE ON CAR GUESSING: KID GUESSING: HAT THE ESP GAME

9 © 2004 Carnegie Mellon University, all rights reserved. Patent Pending.

10 4.1 MILLION LABELS WITH 23,000 PLAYERS THE ESP GAME IS FUN THERE ARE MANY PEOPLE THAT PLAY OVER 20 HOURS A WEEK

11 LABELING THE ENTIRE WEB INDIVIDUAL GAMES IN YAHOO! AND MSN AVERAGE OVER 5,000 PLAYERS AT A TIME 5000 PEOPLE PLAYING SIMULTANEOUSLY CAN LABEL ALL IMAGES ON GOOGLE IN 30 DAYS!

12 A FEW MILLION LABELS CAN IMPROVE IMAGE SEARCH CAN BE USED TO IMPROVE COMPUTER VISION

13 SINGLE PLAYER GAME A SINGLE PERSON CAN PLAY WITH PRE- RECORDED ACTIONS AS THEIR PARTNER WHEN 2 PEOPLE PLAY, WE RECORD EVERY ACTION WITH TIMING INFORMATION WE EMULATE PARTNER BY PLAYING PRE- RECORDED MOVES (0:12) CAR (0:15) HAT (0:21) KID (0:08) BOY (0:23) CAR NOTICE THAT THIS DOESN’T STOP THE LABELING PROCESS!

14 WHAT ABOUT CHEATING? IF A PAIR PLAYS TOO FAST, WE DON’T RECORD THE WORDS THEY AGREE ON

15 WE GIVE PLAYERS TEST IMAGES FOR WHICH WE KNOW ALL THE COMMON LABELS: WE ONLY STORE A PLAYER’S GUESSES IF THEY SUCCESSFULLY LABEL THE TEST IMAGES WHAT ABOUT CHEATING?

16 SEARCH RESULTS

17 SAMPLE LABELS BEACH CHAIRS SEA PEOPLE MAN WOMAN PLANT OCEAN TALKING WATER PORCH

18 SADDAM MR. WILSON MAN FACE MOUSTACHE SAMPLE LABELS BUSH PRESIDENT DUMB YUCK COMING SOON: MEET YOUR SOUL MATE THROUGH THE GAME!

19 WHY DO PEOPLE LIKE THE ESP GAME?

20 THE ESP GAME GIVES ITS PLAYERS A WEIRD AND BEAUTIFUL SENSE OF ANONYMOUS INTIMACY. ON THE ONE HAND, YOU HAVE NO IDEA WHO YOUR PARTNER IS. ON THE OTHER HAND, THE TWO OF YOU ARE BRINGING YOUR MINDS TOGETHER IN A WAY THAT LOVERS WOULD ENVY. ” THE ESP GAME GIVES ITS PLAYERS A WEIRD AND BEAUTIFUL SENSE OF ANONYMOUS INTIMACY. ON THE ONE HAND, YOU HAVE NO IDEA WHO YOUR PARTNER IS. THE ESP GAME GIVES ITS PLAYERS A WEIRD AND BEAUTIFUL SENSE OF ANONYMOUS INTIMACY. “

21 STRANGELY ADDICTIVE “ ” IT’S SO MUCH FUN TRYING TO GUESS WHAT OTHERS THINK. YOU HAVE TO STEP OUTSIDE OF YOURSELF TO MATCH “ ” IT’S FAST-PACED “ ” HELPS ME LEARN ENGLISH “ ”

22 BLOGS IN OTHER CROWNING ACHEIVEMENTS, I HAVEN'T PLAYED IN AT LEAST A WEEK (PROBABLY LONGER), AND I AM STILL ON THE TOP TEN LIST OF PLAYERS ON THE ESP GAME. NUMBER EIGHT IF I REMEMBER CORRECTLY. “ GOD, MY LIFE IS SAD. ”

23 THERE ARE MANY FASCINATING THINGS ABOUT THE ESP GAME: HOW IT USES A GAME STRUCTURE TO BUILD AN INDEX OF IMAGES ON THE WEB WHAT WORDS BECOME TABOO FROM FREQUENT USE HOW INTERESTING SOME OF THE IMAGES PULLED RANDOMLY FROM THE WEB ARE “ BUT BY FAR THE MOST INTRIGUING ASPECT OF THE GAME IS HOW OFTEN YOUR RANDOM, UNKNOWN PARTNER IS A COMPLETE IDIOT. ”

24 OTHER GAMES

25 LOCATING OBJECTS IN IMAGES THE ESP GAME TELLS US IF AN IMAGE CONTAINS A SPECIFIC OBJECT, BUT DOESN’T SAY WHERE IN THE IMAGE THE OBJECT IS SUCH INFORMATION WOULD BE EXTREMELY USEFUL FOR COMPUTER VISION RESEARCH

26 PAINTBALL GAME PLAYERS SHOOT AT OBJECTS ON THE IMAGE SHOOT THE: CAR WE GIVE POINTS AND CHECK ACCURACY BY GIVING PLAYERS IMAGES FOR WHICH WE ALREADY KNOW WHERE THE OBJECT IS

27 X% OF IMAGES(100-X)% OF IMAGES PAINTBALL GAME DON’T KNOW

28 BALL

29 REVEALING IMAGES REVEALERGUESSER CAR PARTNER’S GUESS BRUSH BRUSH CAR CAR

30

31 MEDIUM-TERM GOAL USE THE ESP GAME, ALONG WITH THE PAINT- BALL GAME AND THE REVEALING GAME TO CREATE THE MOST ACCURATE COMPUTER VISION ALGORITHMS TO RECOGNIZE A WIDE VARIETY OF OBJECT CATEGORIES