The Making-of the Abstract Painters – Part I

In my previous post, I tried to sketch how some decisions concerning this project, before actually starting to work, have been made. Therefore, I knew which technologies I am going to use and I knew that my inspiration is Kandinsky. I also knew that I wanted to bring paintings into life and make them alive!

The vision of doing something that mixes music, painting and art mediation was clear since the beginning, but still it was not clear how all these could be joined into one “environment”.

“Interact with the painting to make music” was the starting point and since Kandinsky has always been a big inspiration to me, I selected the painting “Merry Structure”. The title of this painting fitted totally to my motivation. Build a 3D structure for a VR environment.

Original painting

I was looking at the painting, which was printed out on a piece of a sized A4 paper and was trying to imagine how it would be, if I could jump into the painting.  What would it be if this painting is a room of a house and these triangles, circles or lines are actually furniture in the room? I can move them around. I can place them in the space wherever I want to!

I broke down the whole painting selecting most of the objects and transformed them into 3D objects (of course that was the job of the animation designer). The arrangement of the objects in space vertically and horizontally follow exactly the arrangement of the painting, so that if you are standing on the edge of the room you can recognise the original painting.

VR transformation of the painting

In the next step graphics and interaction (grab and move around) were implemented to the UE4 and I already knew that I am going to love this environment! The moment of putting my VR headset and being in this virtual room was overwhelming! I could move around circles, triangles or lines! However, something was missing! Where is the sound? What kind of sound? Which criteria? What would a user expect to hear?

Then the systematic musicologist in me woke up. Besides being a composer, I have a PhD in Systematic Musicology and specifically on audio-visual interactions, synaesthetic perception and cross-modal correspondences! How could it be different? If Kandinsky is my inspiration, I am automatically aware of topics like synaesthesia or the spiritual in art.

Now it becomes tricky! Doing something for VR means automatically that you must have a visual surrounding in there. Vision is the first thing that counts in VR. Through our experiences from our daily life we expect that things in a VR environment behave exactly as they would behave in the physical world. Whatever breaks these “unwritten” rules, it makes us feel uncomfortable or weird because it is not compatible to our expectation. For example in Kandinsky’s VR environment, the big yellow circle stands almost up on the ceiling. Normally the object, if it’s not hanging from somewhere, should fall down because of gravity, but in a virtual world gravity isn’t a “must”. Of course, you can simulate it but you do not have to. You have such a freedom to create environments that would seem unrealistic to our reality and that is one of the biggest advantages of VR.

Go back to the “tricky” point. What happens with sound? How do I assign sound to specific objects? Is it so that the user expects to hear a low-in-range sound for a big object and a high-in-range sound for a small object? That is correct. We are used in making such correspondences intuitively therefore; I am going to follow this rule. Then the next question pops up: what kind of sounds? Should it be a piano or a classical instrument in general or could be any electronic sound appropriate? I made the artistic decision of using electronic sounds because there is no evident connection between Kandinsky and electronic sounds. I wanted to join two periods, which are very much apart from each other, together.