Introduction

In the dynamic realm of film production, the deliberate selection and synergistic combination of film color and editing are pivotal for crafting compelling narratives, conveying emotions, and influencing the audience’s emotional perception (Alfarraji and Al Smadi 2023; Block 2020; Xing and Zhuang 2022). Masterful filmmakers such as Alfred Hitchcock and Victor Fleming have exemplified this through their iconic works ‘Psycho’ and ‘The Wizard of Oz,’ respectively. Hitchcock’s meticulous shot design and editing in suspense films created a heightened sense of fear and tension in audiences (Wang and Wang 2020), while Fleming’s innovative transition from black-and-white to color crafted a mesmerizing and emotionally unexpected cinematic experience. These examples of using intuitive or experience-based approaches to handle film color and editing demonstrate that color and editing are deliberately designed to shape the audience’s emotional perception. However, beyond practical intuition, a theoretical investigation will help to address the question of how to use them together to convey emotion effectively. The current academic literature supports their respective impacts on emotional perception and response with psychological experiments from theoretical aspects (Calbi et al. 2017; Suk and Irtel 2008). Despite the acknowledged impacts of color and editing on emotional perception individually, systematic experiments examining their combined effects remain scarce, primarily due to a historical focus on their individual film components and the methodological challenges in isolating these variables in authentic films. This oversight in film studies literature underscores the need for our interdisciplinary approach, blending practical filmmaking with theoretical insights from psychology and neurocinematics (Hasson et al. 2008; Tan 2018). The primary aim of our study is to address this gap by investigating the combined impact of film color and editing on emotional perception through behavioral and neural experiments. Both subjective scales and objective fMRI techniques are utilized to assess the subjective and objective aspects of emotions.

Transitioning from black-and-white to color in film during the early 20th century marked a significant evolution in cinema, with black-and-white and color films emerging as two distinct categories (Brown et al. 2013). The strategic employment of color in films serves as a powerful medium for conveying emotions, as evidenced by Steven Spielberg’s use of a singular red color in the predominantly black-and-white ‘Schindler’s List,’ which creates a profound emotional impact. Similarly, Yimou Zhang’s ‘Hero’ utilizes a palette of varying colors to distinguish between different narrative segments, thereby underscoring the emotive power of color in storytelling. The impact of color on the audience’s emotional perception and response has been robustly documented in psychological studies (Suk and Irtel 2010; Takahashi and Kawabata 2018; Wilms and Oberfeld 2018), demonstrating that brighter colors tend to be associated with positive emotions, while darker colors suggest negative ones. Beyond merely augmenting black-and-white, color serves as a unique storytelling medium (Johnson 1966). In contemporary cinema, the artistic and thematic use of black-and-white film, such as in evoking historical authenticity or intensifying specific genres, continues to be significant (Li 2012). The persistent presence and distinctive application of black-and-white film prompt intriguing questions about its comparative impact on audience emotional perception. Emerging research addresses this query. Detenber et al. (Detenber et al. 2000) found that color films often receive higher emotional valence ratings than black-and-white, suggesting they add emotional depth. İyilikci et al. (İyilikci et al. 2023) further explored how sound and color together influence emotions, finding that color films, particularly in neutral settings, secure higher valence ratings than black-and-white films. This comprehensive investigation into the influence of color on audience emotional perception, supported by historical examples and psychological studies, confirms that film color—primarily through variations in saturation for colored films and gradations of gray in black-and-white films—plays a significant role in shaping audience emotional perception.

In the early days of cinema, films were often presented without editing, a practice predating the 20th century. Pioneers like Georges Méliès and Edwin S. Porter were some of the earliest to recognize the potential of editing techniques (Payri 2022; Sadoul and Templin 1947), acknowledging their capacity to forge narrative continuity and bolster emotional resonance. The study of film editing’s impact on emotional perception is anchored in early 20th-century montage theory, as exemplified by the Kuleshov experiment (Prince and Hensley 1992). This groundbreaking study, which contrasted an actor’s neutral expression with various emotional scenes, demonstrated that viewers attributed different emotions to the actor based on the surrounding scene. For example, in a happy editing condition, when the emotional scene features a little girl playing with a doll, the viewer perceives happiness in the Russian actor Ivan Mozhukin’s neutral face. While the original footage of this experiment is no longer available, its essential principles have been validated by recent research on black-and-white film sequences (Barratt et al. 2016; Calbi et al. 2017). These modern studies used neutral faces from the Karolinska Directed Emotional Faces and emotional scenes (fearful, neutral, happy) from YouTube, editing them together through continuity editing, where shots are compiled to form a seamless narrative. This resulted in a face-scene-face sequence, requiring participants to assess the emotional valence of neutral faces. They found that the valence results depended on the type of emotional scene edited. As an illustration, when editing a fearful scene into the sequence, participants perceived fear in the neutral face. Further research, including studies incorporating color pictures and videos, has continued to explore the impact of film editing on emotional perception using the scene-face sequence (Ildirar and Ewing 2018; Mobbs et al. 2006; Zheng et al. 2022). Uniformly, findings from color film sequences and black-and-white sequences affirm that film editing profoundly affects emotional perception in viewers.

However, while exploring the individual effects of film color and editing on emotional perception contributes valuable insights (Calbi et al. 2017; Suk and Irtel 2008), this approach does not capture the complexity of their combined impact on audience emotions. Creating an integrated film inherently merges color and editing (Cartwright 2012; Honthaner 2013), leading viewers to perceive the film as a cohesive whole, an idea supported by Gestalt theory (Fahlenbrach 2008; O’Connor, 2015). Furthermore, the emotional perception and response are not only influenced by these elements but also by the collective impact of visual and auditory stimuli, suggesting a more complex interaction between color, editing, and other sensory inputs (Chapados and Levitin 2008; İyilikci et al. 2023; Müller et al. 2012). Moreover, psychological research highlights that different colors can trigger varied emotional reactions in humans, and film editing, by altering narrative pace and visual presentation, can further amplify these effects (Rumyan 2019; Valuch and Ansorge 2015; Young et al. 2013). These factors suggest a synergistic effect when combining these elements, potentially molding the audience’s emotional responses more profoundly. This study undertakes a behavioral experiment using the subjective valence rating scale to examine how film color and editing together shape emotional perception from neutral faces. The goal is to provide insights that enable filmmakers to integrate these elements more effectively in the early stages of production, thereby enhancing the emotional impact on the audience.

While quantitative assessments using scales provide valuable insights, it is essential to recognize the inherent limitations of this subjective rating methodology. Functional Magnetic Resonance Imaging (fMRI) complements these subjective assessments by analyzing the direct neural responses of viewers to film elements. It underscores its importance by providing a method for assessing the consistency of group brain activation in evaluating filmmaking effects (Hasson et al. 2008) and by deepening the understanding of the neural mechanisms behind continuity editing (Magliano and Zacks 2011). This methodology is essential for probing the varied neural responses to the combined effects of film color and editing. Research focusing on the neural basis of color and editing separately, such as studies revealing activations in key brain regions like the occipital lobe, caudate, anterior insula, and fusiform gyrus for color (Liu et al. 2019; Nakajima et al. 2014; Simmons et al. 2007), and the precuneus, insula, temporal pole, anterior cingulate cortices (ACC), and amygdala for editing (Magliano and Zacks 2011; Mobbs et al. 2006), indicates that the synergistic impact of these elements on emotional perception, particularly how these elements elicit distinct activations, has yet to be fully explored. We aim to delve into a comprehensive neurocinematic analysis of how these film elements collectively influence the viewer’s emotional perception. By focusing on observing diverse brain activation patterns for various color and editing combinations, especially in regions associated with emotions, such as the insula and the ACC (Hutcherson et al. 2005; Sonkusare et al. 2023), and evaluating the dissimilarity of these patterns, we intend to unravel the interactions between these film elements through objective measurements.

In the current study, we aim to investigate the interaction between film color and editing by assessing viewers’ emotional perceptions using subjective scales and neurobiological responses with fMRI. To achieve this goal, we selected color and black-and-white as two distinct levels of film color, recognizing them as pivotal in influencing viewers’ emotional responses (Detenber et al. 2000). Similarly, we utilized the shot-reverse-shot structure combined with a face-scene-face sequence paradigm for film editing. This technique, which is prevalent in cinematic storytelling, has been proven effective in influencing viewers’ emotional perception (Barratt et al. 2016; Bordwell et al. 1993). In Experiment 1, we examine how film color and editing interact to affect viewers’ subjective assessments of emotional perception, as measured by valence ratings. In Experiment 2, we investigate whether fMRI techniques can reveal an interaction between film color and editing, demonstrated by distinct brain activation patterns across the brain or within regions related to emotional perception. Through these experiments, we seek to uncover the behavioral and neural responses to filmmaking elements that underpin film reception, while also comparing different editing and color conditions to provide actionable insights for filmmakers.

Methods

The study’s experimental procedures were approved by the Ethics Committee of the State Key Laboratory of Cognitive Neuroscience and Learning at Beijing Normal University (Approval No. IRB B 0030 2019001). The procedures used in this study adhere to the tenets of the Declaration of Helsinki.

Participants

We recruited a cohort of 208 healthy individuals (105 females), aged 17–40 (mean age: 22.64 ± 3.75 years), with normal or corrected-to-normal vision from Beijing Normal University. Recruitment spanned from October 10, 2019, to September 16, 2023, to ensure a diverse and adequately representative sample. A pre-screening questionnaire was used to determine eligibility, specifically to rule out any history of panic disorder. Individuals with a background in film studies were excluded to minimize potential biases associated with their specialized knowledge influencing emotional perception. All participants provided written informed consent before their participation and received monetary compensation for their contributions to the research.

General design

A two-factor mixed design was utilized to assess the interactive effects of film color (a between-subject factor) and film editing (a within-subject factor) on emotional perception in both experiments. Specifically, film color encompassed both color and black-and-white, while film editing content involved juxtaposing shots of neutral faces with varied emotional scenes (fearful, neutral, and happy) using a face-scene-face structure. This arrangement resulted in six distinct conditions: color & fearful editing, color & neutral editing, color & happy editing, black-and-white & fearful editing, black-and-white & neutral editing, and black-and-white & happy editing. Each participant completed 30 trials, organized into three blocks of 10 trials each.

In the material rating experiment, 24 participants were randomly assigned to either color or black-and-white groups using a computer-generated protocol. 12 participants (6 females) experienced colored faces and scenes, and 12 (6 females) experienced black-and-white faces and scenes. In Experiment 1, the behavioral experiment, 117 participants were randomly assigned to color or black-and-white sequences. 58 (28 females) experienced color sequences, and 59 (29 females) experienced black-and-white sequences. For Experiment 2, the fMRI experiment, 67 participants were randomized to color or black-and-white sequences. 36 participants (19 females) completed color sequence tasks, and 31 (16 females) completed black-and-white sequence tasks.

Stimuli

To align with our experimental design, we created film clips for six distinct conditions using a shot-reverse-shot structure, as existing materials with complex mise-en-scène did not meet our objectives. The current method juxtaposes an actor’s neutral expression (the “shot”) with a corresponding emotional scene (the “reverse shot”) to simulate the actor’s perspective effectively. The production process entailed filming the actor’s expressions and emotional scenes in separate sessions. The valence of these materials was evaluated to establish an emotional baseline, which then informed the integration into cohesive film sequences for each condition.

Film materials shooting

Guided by a professional film director and crew, we created materials featuring actors with neutral expressions and various emotional scenes (Fig. 1). The camera setup adhered to the one-sided 180-degree axis rule, ensuring that face shots and emotional scenes were viewed from the same vantage point. Camera One (Cam1) recorded thirty 2-s face clips against a blue screen, featuring a gender-balanced selection of actors. Camera Two (Cam2) captured thirty 4-s clips of emotional scenes in accordance with the Average Shot Length (ASL) typical of mainstream Hollywood cinema (Cutting et al. 2011; Salt 1974). These scenes were divided into fearful, neutral, and happy categories, with ten clips for each category. Examples of these are demonstrated in the Supplementary Videos (Supplementary Video S1) or on YouTube (https://youtu.be/0pFlxQjdLZo). Although Cam1 and Cam2 operated separately, the filming was designed as if occurring within the same scene. After that, color correction was carefully applied to each clip to ensure cinematic quality. The face clips and scene clips were retained in full color for the color film sequences and converted to grayscale for the black-and-white sequences.

Fig. 1: Experimental film sequence production.
figure 1

This panel depicts the three-step process of creating film sequences. Initially, two cameras, strategically positioned along a 180-degree axis, were used to capture the sequences using a shot-reverse-shot structure: one camera recorded neutral faces against a blue screen, and the other captured emotional scenes. The face clips and scene clips were retained in full color for the color film sequences and converted to grayscale for the black-and-white sequences. Participants evaluated the valence of both face and scene materials. In the final stage, face clips were digitally extracted, their backgrounds replaced, and then paired with corresponding emotional scenes, resulting in film sequences available in both color and black-and-white formats. This figure was produced by the authors. This figure was produced by the authors. This figure is covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of Yiwen Wang; copyright © Cao et al. all rights reserved.

Full size image

Materials rating

To evaluate the combined effect of film color and editing on emotional perception, we aimed to achieve uniform valence among neutral faces across conditions, thus establishing a consistent baseline. In contrast, we expected significant valence differences across the emotional scene categories.

Randomization procedures were employed to assign color-neutral faces to three conditions and black-and-white neutral faces to another three, ensuring a gender balance among the actors. Twenty-four participants, divided evenly between assessing color and black-and-white, rated the valence of these faces on a scale ranging from −4 to +4. The participants then evaluated the valence of emotional scenes using the same scale. Each rating was allocated a response time of 3.5 s, followed by a 2-s inter-trial interval (ITI) between trials. The process of stimuli delivery and response recording was managed using PsychoPy 3.2 software (https://www.psychopy.org/) on a 14-inch laptop. The ratings in the experiment were obtained using an integrated approach of the conventional valence rating scale (Barratt et al. 2016) and Visual Analogue Scales. The distance between the participants and the screen was set at 60 cm. Participants provided their assessments by clicking the corresponding value on the scale using a mouse. Analysis of the ratings revealed an average valence for neutral faces of 0.02 ± 0.09, with no significant differences across the six experimental conditions (F2.2, 20.5 = 0.952, p = 0.413, η2p = 0.096, Greenhouse-Geisser correction). Notably, significant differences in valence were observed among the three emotional scene categories for both color (F2, 8 = 119.650, p < 0.001, η2p = 0.968) and black-and-white (F2, 8 = 107.922, p < 0.001, η2p = 0.964) film sequences, with detailed rating scores shown in Table 1. Post hoc tests using Bonferroni correction confirmed significant differences in valence among each scene group in both color and black-and-white sequences (p < 0.001).

Table 1 Materials rating results for emotional scenes.
Full size table

Film sequences combing

In the production of color film sequences, we used the shot-reverse-shot technique to pair clips of neutral faces with emotional scenes—fearful, neutral, or happy. We cut a neutral face shot, followed by an emotional scene shot, and then another neutral face shot together, producing ten distinct film sequences for each emotion type. To ensure visual coherence and authenticity (Carroll and Carroll 1993; Smith 2012), actors were digitally extracted from the blue screen background. This was followed by replacing the blue screen with videos or images captured concurrently with the emotional scenes. Adjustments to the color temperature, brightness, and contrast were carefully carried out to ensure the face clips harmonized with the emotional scenes. Consequently, each film sequence comprised a sequence of a neutral face (Face_1), an emotional scene, and another neutral face (Face_2), as depicted in the right panel of Fig. 1. The completed film sequences were rendered in MP4 format with a resolution of 1920 × 1080 pixels. In line with established research practices (Barratt et al. 2016; Calbi et al. 2017), sound was omitted to concentrate on the visual emotional cues. A comparable procedure was applied to produce the black-and-white film sequences. The Supplementary Video (Supplementary Video S2) or the YouTube link (https://youtu.be/E48tqo8HTNE) demonstrates the final film sequences.

Subjective scales

In the current study, we utilized the valence scale as the primary measure for assessing emotional perception in both experiments. The valence scale was central to our analysis, requiring participants to rate negative, neutral, or positive emotions from a given range. This scale has been widely used in previous studies for assessing the emotional perceptions affected by film color and editing (Barratt et al. 2016; Calbi et al. 2017), establishing it as our main focus.

As the valence scale was an effective indicator for exploring interactions, other emotion scales served as supplementary measures. In Experiment 1, the emotional intensity scale was employed as a secondary measure to assess the subjectively perceived emotional intensity of the actor’s performance. According to Cognitive Appraisal Theory (Richard 1984), a dissociation existed between emotional experiences and physiological reactions, which justified the use of the emotional intensity scale to complement our primary valence findings. In Experiment 2, the arousal scale served as an auxiliary measure to quantify physiological responses to the neutral face, enhancing our understanding of the physiological underpinnings of emotional responses identified primarily through the valence scale.

Procedure

Experiment 1

Participants initiated Experiment 1 with two practice trials, utilizing film sequences distinct from the main experiment (Fig. 2a), to familiarize themselves with the procedure. Each trial commenced with an 8-s film sequence composed of a 2-s neutral face clip, a 4-s emotional scene, and a concluding 2-s display of the same neutral face. After viewing the film sequence, participants evaluated the portrayed neutral face using two scales: the first for valence, ranging from −1 (negative) to 1 (positive), and the second for emotional intensity, from 1 (low) to 5 (high). Each assessment was given a 5-s response window. Trials concluded with a 1.5-s ITI, facilitating the transition to subsequent trials. The experiment consisted of 30 trials, with stimulus delivery and response capture managed by PsychoPy 3.2 software on a 14-inch laptop. The distance between the participants and the screen was set at 60 cm. Emotional conditions were counterbalanced, with half of the participants following the order ‘fearful – neutral – happy,’ and the other half following ‘happy – neutral – fearful.’ The ratings in the experiment were demonstrated using Visual Analogue Scales, and participants were instructed to rate by clicking the corresponding scale value with a mouse.

Fig. 2: Procedure for experiment 1 and experiment 2.
figure 2

a Experiment 1. The experiment was structured using a mixed design, incorporating color as a between-subject factor (encompassing color and black-and-white levels) and film editing as a within-subject factor (including happy, neutral, and fearful editing). The subplot provides an overview of the procedure for Experiment 1, which began with two practice trials. Each trial started with a 2-s display of a neutral face, followed by a 4-s presentation of an emotional scene, and concluded with a 2-s display of a similar neutral face. After the film sequence, participants rated the valence and emotional intensity of the neutral face within a 5-s response time. Each trial ended with a 1.5-s inter-trial interval (ITI). Notably, each condition included ten distinct film sequences. b Experiment 2. The experimental design was similar to that of Experiment 1. The subplot provides an overview of the procedure for Experiment 2, which began with two practice trials. Subsequent to these trials, participants underwent structural imaging before starting the formal fMRI experiment. Each trial consisted of a sequence starting with a 0.5-s crosshair sign, followed by a 2-s display of a neutral face, a variable jitter lasting 4 to 6 s, a 4-s emotional scene, another jitter, and concluding with a 2-s display of a similar neutral face. Then, a 0.65-s inter-stimuli interval (ISI) ensued. After each film sequence, participants were asked to rate the valence and arousal of the neutral face within a 5-s window. Each trial concluded with an ITI of 1 to 1.5 s. Each experimental condition comprised ten trials. This figure was produced by the authors. This figure was produced by the authors. This figure is covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of Yiwen Wang; copyright © Cao et al. all rights reserved.

Full size image

Experiment 2

Experiment 2 included procedural modifications to differentiate blood-oxygen-level-dependent (BOLD) activity between neutral faces and emotional scenes (Mobbs et al. 2006). These modifications entailed the introduction of randomized jitters, characterized by variable timing intervals, between neutral faces and emotional scenes (Fig. 2b). Such randomized jitters enhance the temporal resolution of event-related fMRI designs through variability in stimulus presentation, thereby aiding in a more precise estimation of neural responses to distinct stimuli.

The experiment began with two practice trials utilizing distinct film sequences, followed by T1-weighted image scans, and then the formal fMRI experiment. Each trial started with a 10-s instruction display, a 0.5-s fixation cross, a 2-s neutral face presentation, random jitters lasting 4 to 6 s, a 4-s emotional scene, another set of jitters lasting 4 to 6 s, and a final 2-s neutral face display. After each film sequence, participants had a 0.65-s inter-stimuli interval (ISI) before rating the portrayed neutral face’s valence and arousal. Ratings were provided on a −4 (negative) to 4 (positive) scale for valence and a 1 (low) to 9 (high) scale for arousal, each within a 5-s response period. A 1 to 1.5-s ITI marked the end of each trial. This procedure was repeated for a total of 30 trials. In the experiment, participants lay inside an MRI scanner and watched films that were projected onto a mirror. The computer, which facilitated the film projection, was positioned approximately 50 cm away from the mirror, whereas the distance from the mirror to the participants’ eyes was between 10 and 15 cm. Stimulus presentation and response recording were conducted using PsychoPy 3.2 software, with the order of film sequence presentations randomized for each participant within each condition. The order of the three conditions was also randomized. The ratings in the experiment were demonstrated using Visual Analogue Scales, and participants were instructed to rate by clicking the corresponding scale value with fMRI-compatible keyboards. Participants first used the two keys on the left-hand keyboard to select the value, and then they used the single key on the right-hand keyboard to confirm their selection.

MRI data acquisition

A Siemens 3 T Prisma MRI scanner was utilized in Experiment 2. T1-weighted data were acquired using Magnetization Prepared Rapid Acquisition Gradient-echo (MPRAGE) imaging, with parameters set to TR/TE/TI = 2530 ms/2.27 ms/1100 ms, flip angle (FA) = 7°, field of view (FOV) = 256 × 256 mm2, number of slices = 208, slice thickness = 1 mm, voxel size = 1 × 1 × 1 mm3. Task-related fMRI scans were performed using a T2-weighted echo planar imaging (EPI) sequence, with TR/TE = 2000 ms/34 ms, FA = 70°, FOV = 200 × 200 mm2, matrix size = 100 × 100, number of slices = 72, slice thickness = 2 mm, slice gap = 0 mm, voxel size = 2 × 2 × 2 mm3, volume number = 480. Fild map parameters were as follows: TR/TE1/TE2 = 720 ms/4.92 ms/7.38 ms, FA = 70°; FOV = 200 × 200 mm2, matrix size = 100 × 100, number of slices = 72, slice thickness = 2 mm, slice gap = 0 mm, voxel size = 2 × 2 × 2 mm3, volume number = 1. The average duration for structural and functional imaging was 25 min and 12 s.

Data analysis

Behavioral data

In both Experiment 1 and Experiment 2, a repeated-measures analysis of variance (ANOVA) was used to examine the interaction between film color and editing on valence. When Mauchly’s Test of Sphericity indicated a violation of the sphericity assumption, the Greenhouse-Geisser correction was employed to adjust the degrees of freedom and calculate the corresponding p-value. Post hoc comparisons among the means were conducted using a Bonferroni correction to account for multiple comparisons. Additionally, a simple effect analysis with a Bonferroni correction was conducted to compare film color across different film editing conditions and vice versa. To ensure consistency, the same analysis was applied to assess supplementary measures of emotional intensity in Experiment 1 and arousal in Experiment 2.

Preprocessing of fMRI data

In Experiment 2, we preprocessed the brain imaging data from the 67 participants using SPM12 software (https://www.nitrc.org/projects/spm/). The preprocessing steps included: 1) Converting scanned DICOM format data to NIFTI format; 2) Performing time-slice correction to rectify time differences between different slices; 3) Conducting field map correction to address geometric distortions caused by magnetic field inhomogeneity, using two different echo times (TE) of 4.92 milliseconds and 7.38 milliseconds, with distortion correction executed using phase and magnitude images; 4) Implementing head motion and distortion correction by setting a motion correction quality parameter at 0.9 and a 4 mm separation, along with Gaussian kernel parameters (FWHM: 5 mm × 5 mm × 5 mm); 5) Registering structural and functional images using normalized mutual information (NMI) as the cost function and setting spatial separation parameters and Gaussian kernel parameters to FWHM 7 mm × 7 mm × 7 mm; 6) Segmenting T1-weighted images into different tissue types, including scalp, skull, cerebrospinal fluid, gray matter, and white matter; 7) Aligning functional images to the Montreal Neurological Institute (MNI) standard template and resampling them to a voxel size of 2 mm × 2 mm × 2 mm; 8) Applying spatial smoothing using Gaussian kernel parameters (FWHM: 4 mm × 4 mm × 4 mm) to enhance the signal-to-noise ratio for group-level analysis.

GLM analysis of fMRI data

In our study examining the combined impact of film color and editing on emotional perception through objective measurements, we focused on the brain activity of a specific facial expression, hereafter referred to as ‘Face_2’ (as shown in Fig. 2b). ‘Face_2’ represents the second presentation of a neutral face following the emotional scene. Previous research has demonstrated that facial emotional perception is influenced by the preceding emotional context within a film sequence (Calbi et al. 2019; Mobbs et al. 2006) and that color can affect emotional perception from faces (Liu et al. 2019). Consequently, ‘Face_2, influenced by the complex interplay of film editing and color, was chosen as an ideal focal point for our investigation.

To assess the specific neural response to ‘Face_2’, we employed a general linear model (GLM) for first-level fMRI data analysis for each participant, employing methodologies akin to those used in various cognitive tasks (Kanjlia et al. 2021; Y. Li et al. 2014). Using the canonical hemodynamic response function (HRF), the GLM modeled BOLD signal changes specifically elicited by the presentation of ‘Face_2’ compared to non-presentation timepoints. The design matrix coded the presentation of ‘Face_2’ as ones and all other timepoints as zeros, effectively contrasting the BOLD responses to ‘Face_2’ against the baseline and facilitating a precise assessment of neural responses to ‘Face_2’. Each experimental condition comprised ten trials; to enhance statistical reliability, we averaged the brain activation patterns across these trials for each condition, resulting in standardized brain activation maps for ‘Face_2’.

These brain activation maps were then subjected to second-level analyses (one-sample T-tests) to determine the main effect of each emotional condition. The results of this analysis were displayed in spm_T maps. For further statistical examination, we utilized xjView software (https://www.alivelearn.net/xjview/), which was chosen for its effectiveness in identifying positively activated brain regions. We applied a cluster threshold of 5, including family-wise error (FWE) correction for multiple comparisons, with the significance level of FWE set at p < 0.05, ensuring the robustness and reliability of our findings.

Additionally, we calculated the correlation between conditions using the entire brain map with FWE correction to explore the consistency of activation patterns across different film conditions. To further elucidate whether film color or film editing exerts a more substantial impact on emotional perception, we defined color similarity (Scolor) and editing similarity (Sediting). A ratio index was then calculated to indicate the relative importance of color versus editing, defined as:

$${Ratio}={S}_{{color}}\,/\,{S}_{{editing}}$$

Where Scolor represents the correlation between identical editing conditions but different colors, and Sediting denotes the average correlation of editing conditions within the same color category. A ratio above 1 implies that color differences cause similar neural activations, highlighting editing’s more substantial influence on emotional perception. Conversely, a ratio below one shows that various editing styles yield similar activations, demonstrating that color significantly affects emotional perception by creating distinct brain activation patterns.

Region-of-interest (ROI) analysis of fMRI data

To further explore the combined impact of film color and editing on emotional perception through objective measurements, we selected brain activation in regions related to emotional perception as biological indicators. Five ROIs represent these biological indicators, each associated with distinct facets of emotional perception. The bilateral ACC is recognized for its role in detecting emotional signals and contributing to emotional regulation (Chechko et al. 2012; Lane et al. 1998), the bilateral insula is involved in integrating emotional perception with the physiological states of the body (Jezzini et al. 2015; Menon and Uddin, 2010; Nguyen et al. 2016), and the right middle frontal gyrus (MFG) is crucial for emotion regulation (Eippert et al. 2007; Ho et al. 2012). Our hypothesis in the current ROI analysis posits that the interaction of color and editing can be detected in these regions related to emotional perception.

In defining these ROIs, we utilized independently verified literature-based coordinates that have been previously identified in fMRI studies as specifically active in response to emotional stimuli. The included regions were the ACC [left (MNI: −4, 26, 34), right (MNI: 6, 24, 30)] (Chechko et al. 2012), insula [left (MNI: −38, −1, −2), right (MNI: 40, −1, −1)] (Nguyen et al. 2016), and the right MFG [MNI: 42, 36, 33] (Ho et al. 2012). A 4 mm radius sphere was centered at these MNI coordinates (Ren et al. 2020), and beta values were then averaged across the voxels within each sphere. To test our hypotheses, we conducted a repeated-measures ANOVA to assess the impact of color and editing on the beta values in each ROI. The statistics of beta values within ROIs were confirmed to be consistent with those observed in the behavioral data analysis, further validating the detection of interactions through objective measurements.

Results

Subjective scales

Experiment 1

In Experiment 1, a 2 (Film color: color, black-and-white) × 3 (Film editing: fearful editing, neutral editing, happy editing) ANOVA showed a main effect of film editing on valence (F1.6, 182.2 = 288.734, p < 0.001, η2p = 0.715, Greenhouse-Geisser correction; Fig. 3a). In addition, a significant interaction between film color and editing on valence was observed (F1.6, 182.2 = 5.739, p = 0.007, η2p = 0.048, Greenhouse-Geisser correction). Although there was no main effect of film color on valence (F1, 115 = 0.117, p = 0.733, η2p = 0.001), color film sequences generally received higher valence ratings than black-and-white sequences in a specific editing condition. Simple effect analysis using Bonferroni correction revealed that under fearful editing, valence ratings for color sequences were significantly lower than for black-and-white sequences (p = 0.024). Conversely, under happy editing, color sequences had higher valence ratings (p = 0.043). Further simple effect analysis using Bonferroni correction showed that film editing significantly influenced emotional perception: under fearful editing, valence was lower compared to neutral editing in both film color conditions (both p < 0.001), while under happy editing, it was higher compared to neutral editing (both p < 0.001). As a supplementary scale, no significant main effect or interaction was found in emotional intensity (Supplementary Fig. S1a).

Fig. 3: Behavioral results.
figure 3

a Averaged valence scores in Experiment 1 revealed a significant interaction effect between film color and editing (F1.6, 182.2 = 5.739, p = 0.007, η2p = 0.048, Greenhouse-Geisser correction). Simple effects analysis showed that color sequences led to significantly greater absolute valence in fearful and happy editing than black-and-white sequences. Additionally, significant differences were observed between each film editing condition (all p < 0.001). b Averaged valence scores in Experiment 2 demonstrated the main effect of film editing on valence (F1.4, 91.2 = 168.000, p < 0.001, η2p = 0.721, Greenhouse-Geisser correction). Error bars represent the standard error of the mean. Asterisks denote significant differences between conditions. (*p < 0.05, ***p < 0.001).

Full size image

Experiment 2

In Experiment 2, a 2 (Film color: color, black-and-white) × 3 (Film editing: fearful editing, neutral editing, happy editing) ANOVA showed a main effect of film editing on valence (F1.4, 91.2 = 168.000, p < 0.001, η2p = 0.721, Greenhouse-Geisser correction; Fig. 3b). Subsequent post hoc tests using Bonferroni correction confirmed significant differences in valence among each condition (p < 0.001), suggesting that film editing can significantly influence emotional perception in specific contexts. With a limited sample size, the analysis did not show a significant interaction between film color and editing regarding valence (F1.4, 91.2 = 2.228, p = 0.129, η2p = 0.033). For the supplementary arousal scale (Supplementary Fig. S1b), a significant main effect of film editing was observed (F1.7, 107.5 = 29.899, p < 0.001, η2p = 0.315, Greenhouse-Geisser correction).

Viewers’ brain activation

Impact of color and editing on entire brain activation patterns

The analysis revealed distinct activation of “Face_2” across the whole brain in each film condition. The activation map for each condition is illustrated in Fig. 4, and the specific activation regions are detailed in Supplementary Table S1.

Fig. 4: Brain activity associated with Face_2 in each condition.
figure 4

Brain activity for the neutral expression, referred to as Face_2 (p < 0.05, FWE-corrected, cluster size > 5 voxels), revealed significant activation across the entire brain in each film condition, with variations in specific regions. A correlation matrix, depicted using a chord diagram in the central circle, quantifies the relationships between conditions. For example, the correlation between the activation map associated with the color & neutral editing condition and the activation map associated with the color & happy editing condition is 0.32. The width of each chord represents the correlation value; a wider width indicates a higher value. The chord diagram showed that similarities in brain activation were consistently lower than 1, indicating that each unique combination of film color and editing leads to distinct brain activation patterns.

Full size image

In the color film sequences, particularly in the color & fearful editing condition, we identified 14 clusters with notable bilateral activation in areas such as the insula, bilateral inferior parietal gyrus (IPG), angular gyrus (AG), and the right MFG. The color & neutral editing condition revealed 20 clusters, including activation in the insula, IPG, median cingulate gyrus (MCC), left ACC, and right MFG. The color & happy editing identified 14 clusters featuring bilateral activation in regions like the insula, IPG, superior parietal gyrus (SPG), postcentral gyrus, and the right precuneus.

Conversely, in the black-and-white film sequences, the black-and-white & fearful editing condition showed 25 clusters, encompassing areas such as the insula, IPG, SPG, Rolandic operculum, and the right MFG. The black-and-white & neutral editing condition had 31 clusters with activation in the IPG, supramarginal gyrus, right MFG, and right ACC. The black-and-white & happy editing observed 19 clusters, particularly in the insula, ACC, IPG, right MFG, and the right AG.

To measure whether color and editing have a combined impact leading to distinct brain activation, we calculated the similarity among each pair of conditions. The correlation matrix, represented in the middle chord diagram of Fig. 4 and elaborated upon in Supplementary Table S2, notably showed that similarities in brain activation were consistently lower than 1. This indicates that each unique combination of film color and editing leads to distinct brain activation patterns. We performed further analyses to ascertain the relative importance of film color or film editing by calculating color similarity, editing similarity, and a ratio based on the correlation matrix. Analysis of this ratio reveals that color exerts a more significant influence than editing on the differentiation of brain activation patterns (Table 2). This pattern was observed in four of the six experimental conditions: color & neutral editing, color & happy editing, black-and-white & fearful editing, and black-and-white & happy editing. This analysis underscores that color information elicits more pronounced brain responses than editing information, suggesting color’s paramount role in influencing emotional perception through film.

Table 2 fMRI Results: comparison of color similarity and editing similarity.
Full size table

Impact of color and editing on brain activation within ROIs

In the analysis, we examined neural activity in brain regions related to emotional perception across the six experimental conditions. Bilateral regions of the ACC and insula and the right MFG were selected based on previous studies (Chechko et al. 2012; Ho et al. 2012; Nguyen et al. 2016).

Initially, we explored the combined effect of film color and editing on neural activities, as illustrated by the bar charts within the top-left blue panel in Fig. 5. A 2 (Film color: color, black-and-white) × 3 (Film editing: fearful editing, neutral editing, happy editing) ANOVA showed that an interaction effect was observed in the left ACC (F1.6, 100.9 = 4.172, p = 0.027, η2p = 0.060, Greenhouse-Geisser correction), suggesting that the combined effect of color and editing influenced brain activation in regions related to emotional perception.

Fig. 5: Signal activity in the insula, ACC, and MFG across six experimental conditions.
figure 5

This figure illustrates mean beta values across these conditions within key brain regions, using literature-based coordinates: the ACC [left (MNI: −4, 26, 34), right (MNI: 6, 24, 30)] (Chechko et al. 2012), insula [left (MNI: −38, −1, −2), right (MNI: 40, −1, −1)] (Nguyen et al. 2016), and the right MFG [MNI: 42, 36, 33] (Ho et al. 2012) region, with error bars representing the standard error of the mean. Black asterisks above the bars indicate significant differences within each brain region, while blue asterisks in the radar chart highlight significant interaction effects, as seen with mean beta values in left ACC (significant interaction effect of film color and editing; F1.6, 100.9 = 4.172, p = 0.027, η2p = 0.060, Greenhouse-Geisser correction). The MNI coordinate system refers to the Montreal Neurological Institute standard brain. (*p < 0.05, **p < 0.005, ***p < 0.001).

Full size image

Subsequently, we compared activation patterns between the color and black-and-white film sequences to check the main effect of color, as illustrated by the radar chart in Fig. 5. Mean beta values in the right insula were higher in black-and-white films than in color sequences, suggesting stronger brain activation. Conversely, mean beta values in the left ACC and right MFG were higher in color sequences. However, the right insula showed a marginally significant color main effect (F1, 65 = 3.408, p = 0.069, η2p = 0.050). In the subsequent analysis using Bonferroni correction, in the right insula, mean beta values were significantly higher in the black-and-white & happy editing compared to the color & happy editing (p = 0.035), indicating that black-and-white films elicited greater activation in the right insula during happy editing. Conversely, in the simple effect analysis of the left ACC using Bonferroni correction, mean beta values were higher in the color & neutral editing compared to the black-and-white & neutral editing (p = 0.005), suggesting color films elicited greater activation in the left ACC during neutral editing.

Lastly, to explore the editing effect, we found a main effect of editing in the left ACC (F1.6, 100.9 = 4.211, p = 0.026, η2p = 0.061), right ACC (F1.6, 105.4 = 5.187, p = 0.010, η2p = 0.076), all with Greenhouse-Geisser corrections. A marginal effect was noted in the right MFG (F1.6, 106.1 = 2.708, p = 0.082, η2p = 0.040, Greenhouse-Geisser correction). Post hoc tests using Bonferroni correction revealed that the beta value in the right ACC during neutral editing was significantly higher than during fearful editing (p = 0.019). For the left ACC, simple effect analysis showed higher beta values in color & neutral editing compared to both color & fearful editing (p = 0.013) and color & happy editing (p < 0.001). Additionally, for the right insula, simple effect analysis using Bonferroni correction demonstrated that beta values were marginally higher in color & neutral editing compared to color & happy conditions (p = 0.097). These findings suggest that the bilateral ACC and right insula serve different yet complementary roles in processing film editing effects.

Discussion

In the world of filmmaking, diverse groups of professionals, such as directors, artists, and editors, work together to produce films that have a profound emotional impact on their audiences. While the narrative power of film color and editing is often harnessed through intuitive or experience-based methods during production, the theoretical underpinnings guiding the use of these elements tend to derive from isolated theories on color or editing (Bordwell 1972; Elliot and Maier 2014). These theories aim to link filmmaking techniques directly with the audience’s emotional perception. However, these isolated theories leave unanswered the question of whether a combined effect of these elements on emotional perception exists. Systematic experimentation to explore this combined effect will help to address the theoretical question, offering new insights into the synergistic influence of color and editing on film reception.

In the current study, we created film sequences using different film color and editing elements and conducted behavioral and fMRI experiments to investigate the interaction of film color and editing on emotional perception. In Experiment 1, an interaction between film color and editing was observed in viewers’ subjective valence ratings. In Experiment 2, the measurement of viewers’ brain activation across conditions highlighted how different combinations of film color and editing evoke varied neural responses. Additionally, when the exploration focused on the left ACC ROI, an interaction between film color and editing was observed in the beta values. These results suggest that through subjective scales and objective fMRI measurements, film color and editing cohesively affect emotional perception. Through this study, our findings not only bridge a critical theoretical gap in film studies but also provide practical insights for filmmakers on how to harness these elements for enhanced emotional engagement more effectively.

Interaction revealed in behavioral and ROI analysis

In the traditional film industry workflow, decisions about color themes and palettes, including the choice between color and black-and-white, are typically made during pre-production. During film production, black-and-white films require specific cinematographic lighting and camera settings that are distinct from those used in color films, specifically tailored to enhance monochrome visuals. Editing and the final color grading, which involves adjustments like modifying brightness and contrast, take place in post-production. If there were no significant interactions between color themes and editing choices affecting the narrative and emotional impact, the conventional workflow might suffice. However, the interaction effects we have observed suggest a need for a deliberate, integrated pre-production plan to align color and editing strategies, maximizing the desired emotional perception from viewers. This approach ensures the effective enhancement of both elements throughout the filmmaking process.

Specifically, through subjective scales and objective fMRI measurements, our results demonstrated significant interactions between film color and editing, notably within the valence score of Experiment 1 and the beta values in the left ACC during Experiment 2. While practical filmmaking often relies on intuitive or experience-based methods from traditional workflows for applying film color and editing, theoretical insights from Gestalt theory could guide filmmakers in selecting the most effective combinations of these elements. According to Gestalt theory (Fahlenbrach 2008; O’Connor 2015), viewers perceive the film as an integrated whole, suggesting that its elements should collectively influence their experience. Additionally, reconceptualizing editing as a contextual factor has shown an interaction between color and context (Codispoti et al. 2012; Suk and Irtel 2006), supporting the observed combined effect in films. Previous studies often focused on isolated aspects of sensory processing (Nakajima et al. 2014; Valuch and Ansorge 2015) without fully exploring the synergistic effect on emotional perception. Our research fills this gap by investigating how film color and editing together affect emotional perception, thereby reinforcing Gestalt theory and providing empirical evidence of the impact of two filmmaking elements on viewer perception.

The behavioral and fMRI differences between each condition

Although we detected an interaction of film color and editing, the conclusion may not yet directly translate into practical insight into filmmaking. Further exploration of emotional perception in each condition can help explain the reasons behind the differences between each condition, thereby suggesting how these observed effects could be integrated into or modify existing color and film theories. In summary, each condition led to a unique impact, as evidenced in the valence scale and neural responses between each condition in our study, as depicted in Fig. 3 and Fig. 5. Additionally, the unique impact was further supported by the correlation values for each film combination, which are less than 1 (Supplementary Table S2), highlighting the importance of analyzing the nuanced differences within each pair of conditions.

As for film editing, significant variations in emotional perception were observed across both color and black-and-white groups (Fig. 3a, b), mirroring findings with black-and-white images (Barratt et al. 2016; Calbi et al. 2017). These results broaden our understanding of editing’s influence on the emotional perception of color films. Additionally, editing context affected arousal levels, with fearful and happy edits inducing greater arousal than neutral (Supplementary Fig. S1b), corroborating research on editing’s emotional impact (Calbi et al. 2017). The significant beta values between editing conditions (Fig. 5) underscore the ACC and MFG’s vital roles in processing and monitoring emotional responses to editing (Lane et al. 1998; Sabatinelli et al. 2011), illuminating the sophisticated relationship between film editing techniques and viewer emotional engagement.

Film color is the most apparent factor to viewers. Studies have revealed that color leads to positive emotions, while black-and-white leads to negative emotions (Kaya and Epps, 2004; Suk and Irtel 2010). However, when film color is mixed with different emotional film editing, the color effect conclusion could be more complex. Our study found that nuanced differences emerged in specific editing contexts (Fig. 3a); under fearful editing, color sequences elicited lower valence than black-and-white, demonstrating color’s ability to amplify negative emotions. Conversely, color sequences in happy editing contexts exhibited higher valence, suggesting color’s potential to enhance positive emotions. These findings support the notion that color can significantly alter emotional valence, more so than black-and-white (Wilms and Oberfeld 2018), even in unpleasant contexts (Codispoti et al. 2012). Interestingly, despite aligning behavioral data with expected emotional responses, the neural activations presented a contrasting scenario: black-and-white conditions activated more clusters than color conditions in both fearful and happy editing contexts (Fig. 4 and Supplementary Table S1). ROI analysis revealed higher mean beta values for black-and-white in the right insula and right ACC in happy editing conditions. Specifically, the right insula, crucial in emotional experiences (Hutcherson et al. 2005; Jezzini et al. 2015), showed increased activation for black-and-white films under happy editing. This divergence between behavioral valence outcomes and neural activations suggests that while color films lead to higher valence scores, black-and-white films may engage more brain regions associated with processing emotions. This potentially indicates more complex emotional processing not directly captured by valence ratings, offering insights into the subtle impact of color in film editing. The divergence between brain responses and behavioral outcomes will be an important focus of future research.

The path from empirical research to filmmaking insights

The path from empirical research to practical filmmaking insights begins with a deep understanding of audience reactions, which is essential for producing fiction films. Famous movie producers often prepare various versions of a film for test screenings, not just to refine the final version but also to ensure it resonates effectively with audiences and minimizes potential economic losses. Some assess audience responses through questionnaires, while others employ neurocinematic techniques such as electroencephalography (EEG) and electrodermal activity (EDA)(Christoforou et al. 2017; Kaltwasser et al. 2019). These techniques optimize film production by testing different edits to determine which version elicits the most significant changes in audience emotions. Additionally, Hasson (Hasson et al. 2008) utilized fMRI in conjunction with the Inter-Subject Correlation (ISC) analysis method during screenings of Sergio Leone’s “The Good, the Bad, and the Ugly” (1966) to quantitatively assess how various filmmaking styles influence viewers’ neural responses. This methodology offers a quantitative neuroscientific assessment of different filmmaking styles’ impact on viewers’ brains, offering a valuable tool for the film industry to enhance product evaluations.

In the current study, beyond theoretical contributions to emotional perception, the conclusions drawn from testing the effectiveness of key filmmaking elements in shaping viewers’ emotional perception offer practical insights for filmmakers. These insights include:

  • Given the interaction of film color and editing on emotional perception from behavioral and fMRI measurements, it is crucial that film color and editing, as specifically defined and examined within the context of our research, be considered in conjunction during the pre-production of the filmmaking process.

  • In terms of subjective scales, color can significantly enhance emotional valence in both negative and positive contexts more effectively than black-and-white. Moreover, editing techniques, as investigated in our study, can significantly influence viewers’ subjective emotional perception.

  • From a neural perspective, within the context of our research, specific color information elicits more sensitive brain responses than specific editing information (Table 2). This sensitivity underscores the importance of filmmakers thoroughly considering color usage, as evidenced by our research findings.

The process of our study can be summarized as a research protocol for optimizing filmmaking through scientific experiments, as depicted in Fig. 6, demonstrating how rigorous scientific inquiry can directly inform and enhance the art of storytelling by giving practical filmmaking insights. During filmmaking, the production team employs various filmmaking elements to create a film. In the film reception phase, viewers’ emotional responses through subjective ratings and neural responses help us analyze the impacts of specific filmmaking choices, such as color and editing, as explored in our experiments. The relationship between selected filmmaking elements and viewers’ emotional responses is examined through experiments to discern their effect on the audience. The outcomes of this exploration can then inform and enrich the filmmaking process, incorporating both traditional techniques and emerging technologies, exemplified by tools like Sora AI, which can further audience engagement and narrative depth. This approach optimizes audience emotional engagement and embodies the symbiosis of theoretical research and creative filmmaking practice.

Fig. 6: Illustrating the path from scientific experiments in film research to filmmaking insights.
figure 6

The top tier outlines the progression from the various filmmaking elements, such as story, acting, color, and editing, to the viewer’s cinematic experience. Viewers watch films and, in turn, generate data through their subjective behavioral ratings and neural responses. The second tier depicts the research protocol used in this study, indicated by the green box, and links its findings from scientific experiments to broader filmmaking insights. This figure was produced by the authors. This figure was produced by the authors. This figure is covered by the Creative Commons Attribution 4.0 International License. Reproduced with permission of Yiwen Wang; copyright © Cao et al. all rights reserved.

Full size image

Limitations and future directions

There are several limitations in the current study. Firstly, although there is a tendency to observe an interaction between color and editing (p = 0.129) on the valence scale of Experiment 2, the limited participant sample size, comprising approximately thirty participants per color group, restricted our ability to detect this interaction. This limitation may have compromised the robustness and generalizability of our findings. Consequently, it is crucial to expand and diversify the participant pool in future studies to enhance the validity and applicability of the results. Secondly, although we observed a main effect of editing on the arousal scale, there was no significant effect of color on both the arousal and emotional intensity scales. This suggests that the original experimental materials might not have adequately captured the complexities of how color variations affect emotional responses, necessitating further investigation through alternative experiments with different designs and film materials. Thirdly, while our study focused on visual elements, the omission of auditory elements represents a modest limitation, given the significant role that sound plays in enhancing cinematic experiences. Future work should not only integrate sound but also explore its interaction with film color and editing to provide a richer understanding of how these elements synergistically influence the emotional resonance of film. Fourthly, the insula has been reported to be involved in color perception (Howard et al. 1998). Further studies are needed to clarify the exact role of the insula in the interaction between film color and editing. Lastly, the need for prospective validation of the insights derived from our findings cannot be overstated. Future research should aim to apply our conclusions in practical filmmaking contexts, testing their effectiveness in real-world scenarios to solidify the bridge between cognitive research and practical filmmaking applications. Such studies will not only validate our findings but also contribute to the iterative refinement of filmmaking practices based on empirical research insights.

Conclusion

In this study, we examined the combined effects of film color and editing on emotional perception. We utilized a meticulous approach to film and assembled diverse material into film sequences across six distinct experimental conditions. Using the subjective valence scale, our analysis revealed a significant interaction between film color and editing, as evidenced in the valence scale in Experiment 1. Objective fMRI measurements in Experiment 2 revealed distinct neural activation patterns across critical brain regions, including the insula, ACC, and MFG. ROI analysis in the left ACC further underscored this interaction of color and editing. In summary, this study bridges the gap between neurocinematic theory and practical filmmaking by utilizing these two types of measurement, offering insights for enhancing emotional resonance through integrated color and editing strategies. Furthermore, it lays the groundwork for future inquiries into the collective impact of cinematic elements on various aspects of film reception.