Performance Annotation, Research Participant C3. Symphony No. 9 in E minor, "From the New World", Op. 95, B. 178, Antonín Dvořák
Figure 1.2: Carly Vickers 2021.
Gestural Tracing. A method for tracing the musical gesture of musicians through custom Virtual Reality (VR) software
Figure 1.3: Carly Vickers 2021.
Performance View. Gestural tracing integrated into the performance in augmented reality (mock-up) for the audience to view.
Figure 1.4: Carly Vickers 2021.
The Research Meshwork.
Chapter Four: Theoretical and Methodological Frameworks
Figure 4.4: Carly Vickers, 2021.
The Performance Meshwork. The performance meshwork of an orchestra demonstrating one layer of the correspondence between the musicians of the orchestra.
Figure 4.5: Carly Vickers, 2021.
The Conductor as a Knot. The performance meshwork knotting at the point of the conductor, where each of the musicians meets and entangles
Figure 4.6: Carly Vickers, 2020.
Gesture Capture Session Set-up.
Figure 4.7: Carly Vickers, 2020.
Video-cued recall interview set-up.
Figure 4.8: Carly Vickers, 2020
Personal audio recorders for the focus groups.
Figure 4.9: Carly Vickers, 2020.
The set-up for the focus group participants aimed to simulate a concert hall view.
Chapter Five: Stage 01
VISUALISING SCORES
Figure 5.2: Carly Vickers, 2017.
Animation experiments and annotations.
Figure 5.3: Carly Vickers, 2017.
Annotated score for animating.
Figure 5.4: Carly Vickers, 2017.
Graphic notation scroll – Enigma Variations by Edward Elgar.
Figure 5.5: Carly Vickers, 2017.
Graphic notation scroll – Bolero by Maurice Ravel.
Figure 5.6: Carly Vickers, 2017.
Colour analysis of scroll – ‘Mars’ from The Planets by Gustav Holst (1).
Figure 5.7: Carly Vickers, 2017.
Colour analysis of scroll – ‘Mars’ from The Planets by Gustav Holst (2).
Chapter Five: Stage 02
IMAGE AND GESTURE
Figure 5.8: Carly Vickers, 2017.
Practice journal entry.
Figure 5.9: Carly Vickers, 2017.
Vertical performance drawing created with marker while conducting.
Figure 5.10: Carly Vickers, 2017.
Excerpt from ‘Mars’ from The Planets by Gustav Holst. Vertical performance drawing created with marker while conducting.
Figure 5.11: Carly Vickers, 2017.
Vertical performance drawing created with marker. Conducting in a 3/4-time signature.
Figure 5.12: Carly Vickers, 2017.
Vertical performance drawing created by conducting with a marker.
Figure 5.13: Carly Vickers, 2017.
Excerpts from ‘Procession of the Nobles’ by Nikolai Rimsky-Korsakov Vertical performance drawing created with marker while conducting.
Figure 5.14: Carly Vickers, 2017.
Collection of vertical performance drawings created by conducting with a marker.
Chapter Five: Stage 03
ANIMATED GESTURE AND POINT-LIGHT EXPERIMENTATION
Figure 5.15: Carly Vickers, 2017.
Point-light experiments with conducting gesture. Initial experimentation in animating conducting gesture was explored through point-light videos that captured the movement of the conductor’s arms with a torch in a blackened space
Figure 5.16: Carly Vickers, 2017.
These low-fi experiments included one-handed and two-handed point-light videos and the differentiation between the gestural language of the left and right hands through different coloured lighting sources
Figure 5.17: Carly Vickers, 2017.
Bike lights could be positioned on the hand in a number of ways.
Figure 5.18: Carly Vickers, 2018.
Excerpt from ‘Mars’ from The Planets by Gustav Holst. I also experimented with the clarity of the light sources to create more abstract effects, which I saw as reminiscent of the Visual Music style.
Figure 5.19: Carly Vickers, 2018.
Mapping left-hand and right-hand gestures.
Figure 5.20: Carly Vickers, 2018.
Hand positions for conductors.
Figure 5.21: Carly Vickers, 2018.
Excerpt from ‘Procession of the Nobles’ by Nikolai Rimsky-Korsakov. Experimenting with shadows as animation.
Figure 5.22: Carly Vickers, 2018.
Sketches of conducting poses and corresponding expressive intention.
Figure 5.23: Carly Vickers, 2018.
Conducting posture throughout a performance, pencil on trace paper.
Figure 5.24: Carly Vickers, 2018.
Excerpt from ‘Mars’from The Planets by Gustav Holst. I extended the point-light experiments by introducing a light baton that more accurately recorded the rotation of the conductor’s wrist within the animation and still images.
Figure 5.25: Carly Vickers, 2018.
Stills montage from light baton video experimentation.
Figure 5.26: Carly Vickers, 2018.
Thinking through digital renderings of conducting gesture.
Chapter Five: Stage 04
WAYFARING THROUGH DIGITAL MATERIALITIES
Figure 5.27: Carly Vickers, 2019.
Tilt brush experiments in three dimensions.
Figure 5.28: Carly Vickers, 2019.
Conducting emblems for time signatures (1).
Figure 5.29: Carly Vickers, 2019.
Conducting emblems for time signatures (2).
Figure 5.30: Carly Vickers, 2019.
Musical signature from Procession of the Nobles by Nikolai Rimsky-Korsakov.
Figure 5.31: Carly Vickers, 2019.
Musical signature for ‘Mars’ from The Planets by Gustav Holst.
Figure 5.32: Carly Vickers, 2019.
Musical signature vignettes from Festive Overture by Dmitri Shostakovich.
Figure 5.33: Carly Vickers, 2019.
Musical signature from Symphony No. 8 ‘Unfinished Symphony’ by Franz Schubert.
Chapter Five: Stage 05
CUSTOM SOFTWARE DESIGN
Figure 5.34: Carly Vickers, 2019.
360 camera capture of conducting gesture.
Chapter Five: Stage 06
VISUALISING CONDUCTING GESTURE THROUGH DIGITAL MATERIALS
Figure 5.35: Carly Vickers, 2019.
Discussion notes with creative technologist.
Figure 5.36: Carly Vickers, 2021.
Baton in hand within The Conductor's Trace user interface.
Figure 5.37: Carly Vickers, 2019.
Testing the initial two hand trace within The Conductor's Trace user interface.
Figure 5.38: Josh Harle, 2019.
Two-handed trace produced as a composite image. This composite image became a very important tool for demonstrating to others what my ambitions for this visual augmentation in the concert hall were.
Figure 5.39: Carly Vickers, 2019.
Demonstrates the first test of the two-hand capture within VR software. The Initial trace quality had harsh corners at the vertex points.
Figure 5.40: Carly Vickers, 2020.
Example of initial fade tests within The Conductor's Trace user interface.
Figure 5.41: Carly Vickers, 2020.
Applying different colours visually delineated the left and right hand.
Figure 5.42: Carly Vickers, 2020.
Example of initial colour map tests within The Conductor's Trace user interface.
Figure 5.43: Carly Vickers, 2020.
Initial trace weight testing within The Conductor's Trace user interface.
Figure 5.44: Carly Vickers, 2020.
Initial explorations combining varied trace weights and colours to describe conducting gesture.
Figure 5.45: Carly Vickers, 2020. .
Composite video exploring beat patterns.
Figure 5.46: Carly Vickers, 2020.
Composite video exploring expressive gestures.
Figure 5.47: Carly Vickers, 2020.
Composite video exploring Slavonic Dances by Antonin Dvořák.
Figure 5.48: Carly Vickers, 2020.
Experimenting with brush styles in VR.
Figure 5.49: Carly Vickers, 2021.
Still from light baton video experimentation.
Figure 5.50: Carly Vickers, 2020.
Both hands with the planar trace effect.
Chapter Five: Stage 07
SOFTWARE USER INTERFACE
Figure 5.51: Carly Vickers, 2021.
The Conductor's Trace interface – recording.
Figure 5.52: Carly Vickers, 2021.
The Conductor's Trace interface – playback.
Figure 5.53: Carly Vickers, 2021.
The Conductor's Trace interface – archiving.
Figure 5.54: Carly Vickers, 2021.
Reflective practice journal.
Figure 5.55: Carly Vickers, 2021.
Walk through video demonstrating The Conductor's Trace user interface.
Figure 5.56: Carly Vickers, 2021.
The Conductor's Trace interface - trace specification.
Figure 5.57: Carly Vickers, 2021.
Walk through video demonstrating the user interface for selecting trace styles.
Figure 5.58: Carly Vickers, 2021.
The Conductor's Trace interface – presets.
Figure 5.59: Carly Vickers, 2021.
The Conductor's Trace interface – loading and exporting recordings.
Figure 5.60: Carly Vickers, 2021.
The Conductor’s Trace interface – camera position.
Chapter Five: Stage 08
THE VR HARDWARE AND PERFORMANCE SYSTEM
Figure 5.61: Carly Vickers, 2019.
HTC Vive headset and controllers.
Figure 5.62: Carly Vickers, 2021.
Oculus Quest headset and controllers.
Figure 5.63: Felipe Olivares Barreto, 2020.
Oculus Quest controllers were the lightest and smallest at the time of the testing.
Figure 5.64: Felipe Olivares Barreto, 2020.
Oculus Quest headset mounted on a mannequin for use as a gesture capture system.
Figure 5.65: Felipe Olivares Barreto, 2020.
Research participant testing the performance capture system.
Figure 5.66: Felipe Olivares Barreto, 2020.
Research participant testing the sensor range for the performance capture system.
Figure 5.67: Carly Vickers, 2021.
Testing the gestural trace with a 3D projector.
Figure 5.68: Carly Vickers, 2021.
Projector screen diagram.
Figure 5.69: Carly Vickers, 2021.
Translucent scrim diagram.
Figure 5.70: Carly Vickers, 2021.
Simulating an augmented reality view.
Chapter Five: Performance Annotations
Figure 5.71: Carly Vickers, 2021.
Performance Annotation.
Figure 5.72: Carly Vickers, 2021.
Composite image demonstrating how conducting gesture creates the gestural trace.
Figure 5.73: Carly Vickers, 2021.
Musical signature.
Figure 5.74: Carly Vickers, 2021.
Example of an animated trace that fades at short intervals.
Figure 5.75: Carly Vickers, 2021.
Thin ribbon trace.
Figure 5.76: Carly Vickers, 2021.
Planar trace.
Chapter Six: Testing with Conductors
Figure 6.2: Carly Vickers, 2020.
The set-up for the gesture capture session.
Figure 6.3: Carly Vickers, 2020.
Compilation of musical gesture from research participants (C1, C2, C3) for Children’s March by Percy Grainger.
Figure 6.3 (2): Carly Vickers, 2020.
Compilation of musical gesture from research participants (C1, C2, C3) for Children’s March by Percy Grainger.
Figure 6.4: Felipe Olivares Barreto, 2020.
The set-up for the gesture capture session.
Figure 6.5: Felipe Olivares Barreto, 2020.
The set-up for the gesture capture session.
Figure 6.6: Felipe Olivares Barreto, 2020.
Introducing the conductor research participants to the technology.
Figure 6.7: Fitting the hand controllers.
Figure 6.8: System testing.
Figure 6.9: Testing arm spans with the Oculus headset sensors.
Figure 6.10: Video camera heights needed careful adjustment to create composite images.
Figure 6.11: Carly Vickers, 2020, Composite image with research participant.
Figure 6.13: Monitor for visual feedback throughout the gesture capture session.
Figure 6.14: Carly Vickers, 2020, Research participant C1 demonstrating gesture size as correlated with dynamic.
Figure 6.26: Carly Vickers, 2021, Mixing of colours to reflect the mixing tones of instruments.
Gesture Capture Session Recordings
Children's March Percy Grainger, Research Participant C2.
*Google Cardboard Viewer
Academic Festival Overture, Johannes Brahms, Research Participant C2.
*Google Cardboard Viewer
Symphony No. 5 in C minor, op. 67, mov 4, Ludwig van Beethoven, Research Participant C2.
Slavonic Dances, Antonin Dvorak, Carly Vickers.
*Google Cardboard Viewer
‘Procession of the Nobles’ from Mlada by Nikolai Rimsky-Korsakov, Carly Vickers.
*Google Cardboard Viewer
‘Jupiter’ from The Planets, op. 32, Gustav Holst, Research Participant C3.
*Google Cardboard Viewer
Symphony No. 9 in E minor, ‘From the New World’, Op. 95, B. 178, Antonín Dvořák, Research Participant C3.
*Google Cardboard Viewer
Children’s March, Percy Grainger, Research Participant C3.
*Google Cardboard Viewer
Carly Vickers, 2020.
Example of screen feedback in gesture capture session.
*Google Cardboard Viewer
Carly Vickers, 2020.
Example of screen feedback in the gesture capture session.
*Google Cardboard Viewer
Symphony no. 40 in G minor, K. 550, mov 4, Wolfgang Amadeus Mozart, Research Participant C1.
Children's March Percy Grainger, Research Participant C1.
Chapter Seven: Testing with Audiences
Figure 7.3: Carly Vickers, 2020.
Focus group participants view the Symphony Graphique listening guide before the simulated performance.
Figure 7.4: Carly Vickers, 2020.
The audience view for the simulated concert experience with the Symphony Graphique listening guide.
Figure 7.5: Carly Vickers, 2020.
The Conductor's Trace recording for Symphony No. 5 in C Minor, Op. 67, mov. 4 by Ludwig van Beethoven, Research Participant C2.
Figure 7.6: Carly Vickers, 2020.
The audience view for the simulated concert experience of The Conductor's Trace for Symphony No. 5 in C Minor, Op. 67, mov. 4 by Ludwig van Beethoven.
Figure 7.8: Carly Vickers, 2020.
The audience view for the simulated concert experience with the Music Animation Machine listening guide.
Figure 7.9: Carly Vickers, 2020.
The Conductor's Trace recording for Academic Festival Overture, Op. 80, by Johannes Brahms, Research Participant C2.
Figure 7.10: Carly Vickers, 2020.
The audience view for the simulated concert experience of The Conductor's Trace for Academic Festival Overture, Op. 80, by Johannes Brahms.
Figure 7.11: Carly Vickers, 2020.
The Conductor's Trace recording for ‘Jupiter’ from The Planets, by Gustav Holst, Research Participant C3.
Chapter Eight: Design Development and Future Research
Figure 8.2: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with the movement of all players creating signatures.
Figure 8.3: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with the movement of some players demonstrating instrumentation throughout the musical work.
Figu 8.4: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation demonstrating different styles of music, for example a march would create more angular musical signatures than a waltz.
Figure 8.5: Carly Vickers, 2021.
Sketch mock-up of possible performance meshwork created when playing with a soloist at the front of the orchestra.
Figure 8.6: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with colour denoting the primary instrumentation at any one time in the musical work.
Figure 8.7: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with the conductor’s signature colour denoting the mixing of instrumental tone colours.
Figure 8.8: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with colour denoting pitch centres and when they change.
Figure 8.9: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with colour denoting musical structure.
Figure 8.10: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with scale denoting dynamic changes.
Figure 8.11: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with scale denoting expressive contrasts and changes across the ensemble.
Figure 8.12: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with opacity denoting dynamics, here the double bass are louder than the rest of the string section.
Figure 8.13: Carly Vickers, 2021.
Sketch mock-up of participants controlling the augmented reality view with the option to turn the performance annotation on and off.
Figure 8.14: Carly Vickers, 2021.
Sketch mock-up of participants controlling the augmented reality view with the option to change the visual design attributes of the trace.
Figure 8.15: Carly Vickers, 2021.
Sketch mock-up of participants controlling the augmented reality view with the option to change the position of the performance annotation.
Figure 8.16: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with score-based information layered onto the visualisation.
Figure 8.17: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with score-based information layered onto the visualisation.
Figure 8.18: Carly Vickers, 2021.
Sketch mock-up of performance meshwork visualisation with score-based information layered onto the visualisation.
Figure 8.19: Carly Vickers, 2021.
Sketch mock-up of the gestural trace as an object hanging in the concert hall.
Figure 8.20: Carly Vickers, 2021
Sketch mock-up of the gestural trace as an installation on campus at UNSW.
Figure 8.21: Carly Vickers, 2021.
Mock-up of The Conductor's Trace projected onto the exterior of Peter Haimerl’s Bavarian Concert Hall, the real-time visualisation is intended to communicate that the orchestra is in a rehearsal session.
Figure 8.22: Carly Vickers, 2021.
Mock-up of The Conductor's Trace projected on to the exterior of the Walt Disney Concert Hall, the real-time visualisation is intended to communicate that the orchestra is in a rehearsal session.
