The role of reward in visual perception
Supervisor: Dr. Martin Thirkettle
Visual learning refers to the enhanced sensitivity to visually relevant stimuli. Training and experience produce long-lasting enhancements in decision making based on sensory evidence. For instance, visual identification is faster for repeated, compared to new stimuli and this effect is sensitive to the exposure time (Wiggs & Martin, 1998). Seeking reward is also a powerful factor in altering behaviour, both in the short term and longer term, with decisions being biased towards the options with high expected value. Moreover, reward is also known to affect perceptual learning tasks. Indeed, in a visual search task high-magnitude reward feedback after a successful trial results in priming of reward associated visual features (Hickey, Chelazzi, & Theeuwes, 2010). Consistent with this is the observed activity enhancement in retinotopic areas corresponding to the trained visual fields (Schwartz, Maquet, & Frith, 2002). In non-human primates, neural activity in visual areas as low-level as V1 is shown to be modulated by the expected value of stimuli (Serences, 2008).
The aim of the present project is to establish the effect of reward on visual sensitivity by identifying the perceptual thresholds for reward- and neutral- associated stimuli. Specifically, we will utilize Candy Crush players as out test population, who have deliberately given themselves many hours of practice. Candy Crush is a “match three” game where the player swaps 2 adjacent candies among several on the gameboard to make a row or column of at least 3 matching-coloured candies. Importantly, there are also “special” candies with larger board-cleaning abilities that serve as power-ups. These candies have different affective in-game values. Therefore, we decided to compare a neutral candy icon with a rewarding one (this icon is associated with the highest reward value in the game).
During the experiment participants (Candy Crush players and non-players) completed a staircase 2AFC procedure whereby one of the images contained a certain target (target present) and the other didn’t (target absent). There were 4 different targets – 2 Candy Crush targets (rewarding and neutral) and 2 control non-Candy Crush targets (rewarding and neutral). The control targets were a triangle (of neutral value) and a smiley face (of rewarding value). We based our control comparison on the finding that positive facial expressions are recognised faster compared to neutral ones (Lamy, Amunts, & Bar-Haim, 2008), thus allowing us to compare the general reward responsiveness of players as a control reward/Candy Crush reward ratio. We predicted that affective-value improves visual sensitivity such that experienced players will have lower visual threshold for the Candy Crush reward – associated stimulus (i.e. 'Candies' which reward the player in-game) relative to the neutral one.
Figure 1. Sample images of the 4 conditions at 4 different levels of diffeomorphic scrambling.
Experimental stimuli and procedure
Both target present and target absent images were generated of a pool of icons. Images constituted a 7×7 visual array (49 single icons). Both target present and absent images were generated from a random mixture of 17 distractor icons, with the addition of a target icon for the target present images. There were 4 types of targets: Candy Crush rewarding (CC-R), Candy Crush neutral (CC-N) as well as non-Candy Crush rewarding (NCC-R) and non-Candy Crush neutral (NCC-N) for control. To prevent habituation effects, 40 different instances of images were created for each participant – each image instance differed in the number of a particular distractor and their positions in the array. The stimuli were created with Python.
The created images were scrambled to different extent, ranging from level of scrambling 59 (no scrambling) to level 0 (highest scrambling) (see figure 1). Images were scrambled using diffeomorphic transformations (Stojanoski & Cusack, 2014). Diffeomorphic scrambling has advantages over other scrambling methods such as phase, box and texture scrambling because it preserves the basic visual properties of images (spatial frequency, perceptual organisation) intact. The scramble images were created with MATLAB using the publicly available script (Stojanoski & Cusack, 2014).
Candy Crush players (n=5) and non-players (n=6) completed the 4 conditions in a random order. Each condition was presented in a separate block, preceded by instructions of the specific target type for this condition. On each trial, there were 2 images, one target-absent and one target-present shown on either side of a fixation point on which participants were requested to fixate. Participants had to indicate with a button press on a Cedrus RB-530 response box on which side is the image containing the target. Each pair of images was present for 1.5 s. (see figure 2). The starting value of the staircase procedure was level of scrambling 50. Participant’s thresholds were estimated as the mean of the last 6 reversals in the staircase.
Figure 2. Schematic representation of the experimental procedure.
Findings
A one way within-subjects ANOVA showed significant variation in thresholds between the 4 conditions, F(1,42) = 10.92, p<0.01. A post hoc Tuckey test showed that the threshold for the neutral control target (M=57.23) was significantly higher than neutral (M=46.24) and rewarding (M=45.08) Candy Crush targets at p<0.05. No significant difference was observed between the other 3 conditions (figure 3). There was no significant difference between players and non-players in both Candy Crush neutral, t(9)= 0.07, p=0.94 and rewarding targets, t(9)=1.35, p=0.21.
Figure 3. a) Thresholds of players and non-players for the 4 conditions. Error bars represent SEM; b) and c) Thresholds of players for Candy Crush neutral and rewarding targets, respectively. Error bars represent SD
Conclusions
There was a difference between the Candy Crush and control neutral target due to difference in the salience of the targets. However, no difference was observed among the 2 game targets and among the 2 control targets for both players and non-players. Also, the threshold for control rewarding target occupies an intermediary position between that of the Candy Crush targets and control neutral target, suggesting that the affective value associated with it enhances perception. Importantly, as the sample size is insufficient to draw firm conclusions, testing participants will continue after the official end of the internship.
Placement
Over the course of this internship I gained valuable knowledge and skills for conducting research in psychophysics. Those can be divided into 3 main points:
Through discussion with my supervisor and reading psychophysics literature I’ve expanded my knowledge about different aspects of visual function and methodological approaches to measure them as well as the strengths and limitations of those approaches. Moreover, I have learned how perceptual thresholds are defined and the importance of visual stimuli and tasks used to experimentally capture them. While designing the stimuli for the staircase procedure, I have gained perspective on the rationale behind different types of image scrambling (phase, box, diffeomorphic etc.) and how they are used to isolate mechanisms associated with recognising objects. This knowledge was useful when designing the experiment and thinking about the theoretical question that this project aimed to answer.
During the project, my supervisor and I programmed the experiment on Python (using PsychoPy coder, Peirce, 2007). As a result, I acquired valuable Python skills for implementing psychophysical experiments into code. Also, I have gained knowledge on different programming languages as well as their fundamental differences and utility for psychology. I was trusted to find solutions to code errors and fix them. Furthermore, I was responsible for recruiting and testing participants as well as data analysis.
I had the pleasure of working with a supervisor who believed that the quality of science can be improved by making it more transparent which I truly support. This includes specifically stating and making publicly available the rationale, methods and predictions of the experiment and data analysis plan beforehand. As such, all programming codes, experimental materials and participants’ data are publicly available in the Open Science Framework (OSF) at https://osf.io/kqspf/
References:
Hickey, C., Chelazzi, L., & Theeuwes, J. (2010). Reward guides vision when it’s your thing: Trait reward-seeking in reward-mediated visual priming. PLoS ONE, 5(11), 1–5.
Lamy, D., Amunts, L., & Bar-Haim, Y. (2008). Emotional priming of pop-out in visual search. Emotion, 8(2), 151–161.
Peirce, JW. (2007) PsychoPy – Psychophysics software in Python. J Neurosci Methods, 162(1-2):8-13
Schwartz, S., Maquet, P., & Frith, C. (2002). Neural correlates of perceptual learning : A functional MRI study of visual texture discrimination. Proceedings of the National Academy of Sciences, 99(26), 17137–17142.
Serences, J. T. (2008). Value-Based Modulations in Human Visual Cortex. Neuron, 60(6), 1169–1181.
Stojanoski, B., & Cusack, R. (2014). Time to wave good-bye to phase scrambling: Creating controlled scrambled images using diffeomorphic transformations. Journal of Vision, 14((12):6), 1–16.
Wiggs, C. L., & Martin, A. (1998). Properties and mechanics of perceptual priming. Current Opinion in Neurobiology, 8(1), 227–233.