Figure skating and artistic research

[zeit_hexe]

Since I returned to skating after a 25 year break, I really wanted to integrate my skating with my actual profession as a composer and computer musician. I have done a lot of work with gestural tracking, building new instrumental controllers, and interactive/live electronic music. Some colleagues of mine created this motion tracking software and system to use with small robot attached to my boot, and I took all the data and translated this into music.

All of the videos and data was originally just supposed to be training for the machine learning to recognise movement patterns, but when I just started experimenting with what kind of sounds could be generated and the musicality of just this very basic skating I was really amazed and created a musical composition out of it - everything is created and modified based on the skating data.

Of course I plan to work with actual skaters in the future and I realise that the skills in the video are super basic and likely not well executed but I am still so excited that even this small and simple proof of concept has generated such cool results!

 

[DizzyFrenchie]

Since I returned to skating after a 25 year break, I really wanted to integrate my skating with my actual profession as a composer and computer musician. I have done a lot of work with gestural tracking, building new instrumental controllers, and interactive/live electronic music. Some colleagues of mine created this motion tracking software and system to use with small robot attached to my boot, and I took all the data and translated this into music.

All of the videos and data was originally just supposed to be training for the machine learning to recognise movement patterns, but when I just started experimenting with what kind of sounds could be generated and the musicality of just this very basic skating I was really amazed and created a musical composition out of it - everything is created and modified based on the skating data.

Of course I plan to work with actual skaters in the future and I realise that the skills in the video are super basic and likely not well executed but I am still so excited that even this small and simple proof of concept has generated such cool results!

I hope that Yuzuru Hanyu will know about it, because I'm sure that he would be very interested! If he's not already experimenting this sort of sound generation.

 

[Mathematician]

Could you explain how exactly you translated motion data into music? There are only 12 tones yet an infinite amount of possible foot positions. What kind of parameters do you set? What sort of movement would be detected as a single data point? What did the raw data sound like?

 

[zeit_hexe]

Could you explain how exactly you translated motion data into music? There are only 12 tones yet an infinite amount of possible foot positions. What kind of parameters do you set? What sort of movement would be detected as a single data point? What did the raw data sound like?

in this case extremely simple - I took the vector from the gyro and the vector from the accel data and applied those to some music software I was running. There are multiple layers, some are more "connected" to the data than others, because I have the processes more obvious (like the sweeping gestures that come in one some of the jumps, for example) The slow piano movement, for example, is slightly less connected because I used the accel vector to affect the duration of note/re-triggering of pitches so that sometimes more movement resulted in a slowing down of the melody. I didn't want every layer to sound the same.

I also used both gyro and accel because the gyro data captures primarily the movement in relation to my own position and accel captured overall movement - I decoupled them as well so that I could do things like have the note volume driven by accel but the selection and gestures driven by gyro. If they were not descoupled, then the same pitches and gestures would always have the same volume, but this way there is more variety in that regard (but still related to the movement)

I don't use only 12 tones - in one layer I have 21 different pitches that are chosen from. If you mean octave, I am also using technically more than 12 because there are 2 different tuning systems I am using. Although yes, I am using a 12 key per octave keyboard to generate the sound..

 

[Mathematician]

in this case extremely simple - I took the vector from the gyro and the vector from the accel data and applied those to some music software I was running. There are multiple layers, some are more "connected" to the data than others, because I have the processes more obvious (like the sweeping gestures that come in one some of the jumps, for example) The slow piano movement, for example, is slightly less connected because I used the accel vector to affect the duration of note/re-triggering of pitches so that sometimes more movement resulted in a slowing down of the melody. I didn't want every layer to sound the same.

I also used both gyro and accel because the gyro data captures primarily the movement in relation to my own position and accel captured overall movement - I decoupled them as well so that I could do things like have the note volume driven by accel but the selection and gestures driven by gyro. If they were not descoupled, then the same pitches and gestures would always have the same volume, but this way there is more variety in that regard (but still related to the movement)

I don't use only 12 tones - in one layer I have 21 different pitches that are chosen from. If you mean octave, I am also using technically more than 12 because there are 2 different tuning systems I am using. Although yes, I am using a 12 key per octave keyboard to generate the sound..

Interesting, I think I understand the process here. Would be interesting to hear raw vs post-production data in any case.

 

[zeit_hexe]

Interesting, I think I understand the process here. Would be interesting to hear raw vs post-production data in any case.

Thanks - I'm not sure what you mean, but basically it is all the raw data that i am using, just I am applying it in specific ways to musical parameters. Here is an example though of a very 1:1 data to sound generation of the gyroscope data where I think it is very obvious and clear, but you can still here somewhat of a difference in musicality between the elements. It was my first test, and just to see what kind of musicality I could find.

This is not really sonification in the sense that most peopel think of, but I am a composer and so my main interest was creating creatively and aesthetically interesting concepts, not necessarily such a direct translation of the data, if that makes sense - I wanted to find the inherent musicality and possibilities of figure skating movement as a musical controller.

 

[zeit_hexe]

I hope that Yuzuru Hanyu will know about it, because I'm sure that he would be very interested! If he's not already experimenting this sort of sound generation.

I think he did a video game related show?
So this is a lot related to gamification, and composer Rob Hamilton for example was using game avatars to generate/compose music about a decade ago, so its quite possible he does.

I do have some collaborators, and I'm always open to collaborative work and data testing for these kinds of things.
So far since I am just even trying to get the thing to work, I've only really used myself as a test subject, I was honestly not expecting in the least to make any of this original data into music, to be quite honest.

 

[Mathman]

This is a fascinating area of research. One way to study gravity waves is to translate resonance from colliding black hopes and from neutrron stars orbiting black holes into music.

 

[Mathematician]

Thanks - I'm not sure what you mean, but basically it is all the raw data that i am using, just I am applying it in specific ways to musical parameters. Here is an example though of a very 1:1 data to sound generation of the gyroscope data where I think it is very obvious and clear, but you can still here somewhat of a difference in musicality between the elements. It was my first test, and just to see what kind of musicality I could find.

This is not really sonification in the sense that most peopel think of, but I am a composer and so my main interest was creating creatively and aesthetically interesting concepts, not necessarily such a direct translation of the data, if that makes sense - I wanted to find the inherent musicality and possibilities of figure skating movement as a musical controller.

Thanks. Yes, thats what I meant, a 1:1 example of the exact movement and what sort of music is generates.

This is a fascinating area of research. One way to study gravity waves is to translate resonance from colliding black hopes and from neutrron stars orbiting black holes into music.

What sort of useful information about gravity waves would you be looking to gain by translating this collision resonance into music?

 

[Mathman]

What sort of useful information about gravity waves would you be looking to gain by translating this collision resonance into music?

I am not an expert on this aspect of physics (although I have published on the certain related themes of reletivistic cosmological modelling). My understanding is that gravity waves are so close to being undetectable that it is helpful to be able to map what information we can observe into a setting that is more amenable to analysis.

Or maybe the point is just to produce some cool-sounding videos to show to funding agencies.

 

[Mathematician]

I am not an expert on this aspect of physics (although I have published on the certain related themes of reletivistic cosmological modelling). My understanding is that gravity waves are so close to being undetectable that it is helpful to be able to map what information we can observe into a setting that is more amenable to analysis.

Well, they are not really detecting waves, they are measuring discrepancies in light's path-time through two "vacuum" tunnels, which they presuppose is caused by transverse spacetime distortion ("gravity wave"). But it isnt actually a mechanical wave in the classical sense. So basically what I'm saying is, they are already putting the interferometric data through a code (convoluted neural-network?), which by the way I believe is private (could be wrong), to represent it as a visual wave for comprehensibility. So, I dont think changing it into music instead would be different, they're already doing a huge transformation for analysis, and the original detection has nothing to do with how you interpret a "wave".

Or maybe the point is just to produce some cool-sounding videos to show to funding agencies.

1. Is LIGO really lacking funding?

2. Are we sure, that its a good thing, that it isnt?

 

[Mathman]

1. Is LIGO really lacking funding?

LIGO’s funding comes from the National Science Foundation. The NSF’s budget for all basic scientific research is about US$ 7 billion this year, down half a billion from last year. (Congress faces the task of convincing the American taxpayer that non-commercial scientific research is worth it.) There is a lot of competition for this money. Each of the 700 or so projects that the NSF sponsors must persuade the NSF that their project is the most deserving. (I have attended -- and led -- a number of seminars on “how to write a successful NSF proposal.")

I agree with the first part of your post. Interpreting all wave phenomena as music does not always have any particular scientific merit. But we humans do like the sound of a vibrating string.

 

[Mathman]

By the way, what is most intriguing about the OP's project is this: Figure skaters are expected to match movement to music -- and they win or lose competitions according to their ability to do so. But what exactly does this mean, besides timimg your triple Axel to the cymbol crash?

 

[4everchan]

By the way, what is most intriguing about the OP's project is this: Figure skaters are expected to match movement to music -- and they win or lose competitions according to their ability to do so. But what exactly does this mean, besides timimg your triple Axel to the cymbol crash?

There are 4 things I watch for
Yes, you named the most obvious 1 : musical highlights with, let's say a triple axel being timed perfectly with a big chord or percussion etc
2 : Pulse : there are obvious beats in music and some skaters manage to follow them really well... not only dancers but also single skaters. They do not do it at all times, but for instance, you can definitely tell during step sequence if skaters are matching the pulse or not matching it.
3: Musical phrase : as music builds up, some skaters are able to show this in how they use power and glide to match the musical phrase's intensity and variation in intensity.
4 : Musical intent/emotion : most pieces of music will bring emotional reaction to the listener. How well does the skater embodies that through movement on the ice. Some give us big face expression, some do it with their full body, coming from the blade.

 

[Mathman]

^ As I understand the OP's project, she is attempting to come at the question from the opposite direction. Traditionally we say "Here is a piece of music. What should the skater be doing in response?"

But is it possible to ask "this is what figure skaters do on the ice; do the skating movements themselves suggest musical phrasing, pulse, emptionsal content, etc.?"

Does a skater (or choreographer) ever say to him/herself -- I have just created the world's best figure skating program. Now I will begin tne search to find appropriate music to match it?

 

[zeit_hexe]

^ As I understand the OP's project, she is attempting to come at the question from the opposite direction. Traditionally we say "Here is a piece of music. What should the skater be doing in response?"

But is it possible to ask "this is what figure skaters do on the ice; do the skating movements themselves suggest musical phrasing, pulse, emptionsal content, etc.?"

Does a skater (or choreographer) ever say to him/herself -- I have just created the world's best figure skating program. Now I will begin tne search to find appropriate music to match it?

in a sense, yes - but it could also be looked at as a process that is more interactive, organic, and reciprocal. the first step is finding some of the inherent musicality within skating movement, within skating sounds, and within the natural phrasing and shaping that takes place within the process of figure skating.
But part of it is also giving the figure skater themselves an instrument, vehicle, of further expression and perhaps a means of even deeper connection with the sound and music that they are skating to, interacting with, or even generating (if these three things are expressed on a continuum).

 

[Mathman]

By the way (speaking of musical astronomy ) the “music of the spheres” concept goes back to Pythagoras in the sixth century BCE. The famous 16th century astronomer Johannes Kepler was a big fan. Kepler deduced that the notes sung by the planet earth in its lugubrious journeying around the sun were “mi, fa, mi.” This stands (Latin) for “misery, famine, misery” which accounts for the wretchedness of human existence.

That goodness we have figure skating to balance the cosmic scales.

 

[4everchan]

^ As I understand the OP's project, she is attempting to come at the question from the opposite direction. Traditionally we say "Here is a piece of music. What should the skater be doing in response?"

Yes. I got that.

But is it possible to ask "this is what figure skaters do on the ice; do the skating movements themselves suggest musical phrasing, pulse, emptionsal content, etc.?"

I have said it in another thread : i don't see music, i hear it So sure, I am not oblivious that movements could lead to a soundscape but music is more than that... For instance, the rain... the wind in the trees making leaves move... this is all what some people may call music.. i call it it soundscape... It can be soothing. It can be frightening even... but does it have a musical phrase, a structure ? is it organized in a way that's predictable and therefore, in a way that we can choreograph on it ?

Does a skater (or choreographer) ever say to him/herself -- I have just created the world's best figure skating program. Now I will begin tne search to find appropriate music to match it?

I doubt so. Improvisation could happen... but then, trying to find music that would fit perfectly on it ? I am not sure it's worth it. Also, would a skater create that perfect program with pulse in mind ? Because that's something else... Let's say hypothetically that I am skating to 2m40 seconds for a SP. I pick one tempo. It has 1 beat per second. So metronome marking 60. So now, I know there are 140 beats in the choreography. I know its speed and pulse. I can organize a program. Can I find music that will reflect that later on ? Yes. Sure..

I think a good program is not simply inspired by a catalog of movements choreographer may have. It comes from within, after reacting to the emotional content of a piece of music. I am not a choreographer but I have improvised and composed for fun at the piano. There is very little randomness in how that's done That's why I think that it's interesting for research, (what the OP is doing) but I don't see how it could really work in real skating life. There could be work done of course, especially at galas in that sense... minimalist music comes to mind where the structure would be so simple that it may work... but in competition programs, I don't think so.

 

[Mathman]

... Let's say hypothetically that I am skating to 2m40 seconds for a SP. I pick one tempo. It has 1 beat per second. So metronome marking 60....

I once suffered a brief spell where I became frustrated with skaters trying to edit their music to come out to the exact number of seconds required in a four minute thirty second program, using such stratagems as randomly repeating a few bars or splicing in a couple of extra measures from some other song. I searched at the music department of my local bookstore for compositions and recordings that were already exactly 270 seconds long and required no editing. My project never amounted to anything, but I did meet Todd Eldredge there once, thumbing through the classical section, perhaps with the same goal.

There used to be a sort of annual choreography competition and skate-off where a skater would choose a theme and design a program, along with selecting the music that they felt went with the theme. The skater and the choreographer might be the same person, or they might be a team consisting of one skater and one (aspiring) choreographer. One year Sean Rabbitt won (“Caffeine” ), Bebe Liang was second (“Time”). The fun for the audience was that the themes were not announced beforehand, so you had to try to guess from the actual performances.

 

[zeit_hexe]

Yes. I got that.

I have said it in another thread : i don't see music, i hear it So sure, I am not oblivious that movements could lead to a soundscape but music is more than that... For instance, the rain... the wind in the trees making leaves move... this is all what some people may call music.. i call it it soundscape... It can be soothing. It can be frightening even... but does it have a musical phrase, a structure ? is it organized in a way that's predictable and therefore, in a way that we can choreograph on it ?

I doubt so. Improvisation could happen... but then, trying to find music that would fit perfectly on it ? I am not sure it's worth it. Also, would a skater create that perfect program with pulse in mind ? Because that's something else... Let's say hypothetically that I am skating to 2m40 seconds for a SP. I pick one tempo. It has 1 beat per second. So metronome marking 60. So now, I know there are 140 beats in the choreography. I know its speed and pulse. I can organize a program. Can I find music that will reflect that later on ? Yes. Sure..

I think a good program is not simply inspired by a catalog of movements choreographer may have. It comes from within, after reacting to the emotional content of a piece of music. I am not a choreographer but I have improvised and composed for fun at the piano. There is very little randomness in how that's done That's why I think that it's interesting for research, (what the OP is doing) but I don't see how it could really work in real skating life. There could be work done of course, especially at galas in that sense... minimalist music comes to mind where the structure would be so simple that it may work... but in competition programs, I don't think so.

gestural control/movement to create music is an absolutely HUGE field that 100% stems from the way music is made on acoustic instruments.

When you pluck a string on a lyre, you are making a physical gesture (and effort) that causes the string to vibrate, then producing sound waves. There is a ton of research and debate, especially since the advent of computer music (and analog generation of music via electricity i.e., synthesis and tape), into the how we reconcile our perception of sound and the source of the sound.

Computer music opened up the possibility for sound to be created without this physical effort, because electricity generates the necessary energy that was once required by a person's physical effort (not fighting against this either way was one of the topics of my first doctorate research, for example: http://www.ems-network.org/IMG/pdf_EMS15_Aska.pdf) - this became a huge debate in computer music - do we need to re-introduce this perceived gestural effort or not? Is it ok to perform in a laptop orchestra, for example, and just click with the mouse? Is this performance? Connecting sound and visual is absolutely nothing new and nothing unusual.

For this project I of course was never imagining it to be competitive programs, I was hoping to work with figure skaters in an artistic context. But if you read my post above, it is meant to be something reciprocal, not a figure skater creating movement because of "how the music is created" - it could result in converting the skater themselves into an instrument, which affects how they interact, but this is also 100% precedented, as this research has been done in gamified music before, for example (Rob Hamilton's Echo Canyon uses a game avatar as a musical instrument and my The missing piece also does something similar, by connecting the musician with the avatar, thus generating the sound world). What I did and do see, especially after having data from just some small and simple movements, is a musical potential within a lot of figure skating movements and artistic potential for musical and physical expression.

of course the sounds are not 100% generative, because I imposed strict parameters for which pitches were playd back, when, and how, and with what instrument, what volume, etc. I chose to layer specific things at specific times for the dramaturgy of the music.