Starting from zero in VR creation: some thoughts from a composer’s perspective

Zero: how everything started…

When this project started in 2018, I knew few things about virtual reality (VR) and what it is really about! At that time, I only had the vision to make paintings somehow alive and bring them to another dimension! A little bit crazy if you consider that I am actually a composer! But what has music to do with visual arts?

Wassily Kandinsky was impressed by the music and its potential as an art form. In my case it’s the other way around: I am impressed by the power of paintings and sculptures!

From the perspective of a classical composer, time is the inevitable element of music. Musical compositions or musical pieces are nothing more than sound events distributed in time. Ok maybe I’m exaggerating a little bit but for us composers time is in a way the frame in which we tell a story.

In visual arts though time doesn’t exist. The painter doesn’t tell a story in a strict time line. The painter may catch a glimpse of a moment but his frame is the space and not the time. Totally different in the way of creative process, or maybe not?

So the vision was there from the beginning:

• What would happen if paintings become alive and come out of their two-dimensional space?
• What would happen if you add a time dimension to a painting?
• What if Kandinsky could make music with his circles, lines and triangles?
• What if Mondrian could come out of the canvas’ frame?

And that is how the idea to this project was born: the mixture of visual arts and music in one!

One: how to do this?

So since I wanted to combine music and paintings into one, the logical question to pop up next was “how do I do this?”

“How can I add another dimension to a painting?”

I was imaging myself being able to jump into the painting and explore it from inside! What would happen if I am in there? What would happen if I was the one screaming in Munch’s painting “The Scream”? What would happen if I can move things around and throw them away?

And that was the time where the idea of VR came in! Why not to use virtual reality to do so?

All these unlimited possibilities!! I could make up a world of my own that has nothing to do with reality. That is why, it is called virtual reality!!

But how naïve from me to think that I can build up a world of my own in VR! I am a composer! I can make music! I can make sound design! But what about visuals? You can’t have VR without having the space, without having visuals! When coming to drawing and visuals I’m all fingers and thumbs! So I had to ask somebody to help me out with all these graphs and visuals.

And that was the time when Stefanos Papadatos came to collaborate into this project!

So a composer was there and a visual artist was there! Now we can start right??

Two: what to use?

The vision was there but I had to wait a little bit longer before something could actually happen!

So where to start? What to use?

Saying that you want to use VR as a technology to make creative content involves a lot of troubleshooting and of course a lot of decision making:

• What do I have to use?
• Should I use Unity or should I use Unreal?
• Should I use Vive or should I use Oculus or should I use Valve Index?
• What is reliable in long-term?
• What about the music?

The main idea was to present paintings in a 3D environment and make music with them! “Make music” means automatically that this VR environment is going to be interactive. So something extra to think about!

After a lot of discussions and a lot of troubleshooting, we came to the conclusion that for the project the best option is to use Unreal Engine and Vive Pro from HTC. On this I am going to explain more in another thread!

Three: starting from scratch

So we have the staff, and we have the technology! We made some decisions!

And now, which is the story that I want to tell?

Next question: which paintings then could work out?

Kandinsky has been a big inspiration to me since many years so the first artist was there! Let’s make Kandinsky 3-D! But then, which painting? It isn’t that Kandinsky painted only a few artworks. And from which era? His early paintings from the Blue Rider period or the ones from the period where he was in Bauhaus?

Here the decision was pretty much easy but due to practical reasons of course. What easier than having to build a 3D world based on geometrical forms rather than abstract and not easy-to-define structures? Imagine that I would select a painting from Monet where the lines don’t separate from each other and the transitions between colours aren’t easy to recognize. Maybe later but not at this point! It’s not that I want to develop something new. In the bottom line I want to use software and hardware that already exists and not to re-invent the wheel! So we have to explore our possibilities with the decisions that we made on the software and on the hardware!

So which painting? I decided to use the painting “Merry Structure”, not the most famous painting by Kandinsky. But the painting itself consists of geometrical objects and it is clear and simple. A good painting to start the experimentation in VR.

So what should happen with the painting? I need a kind of scenario!

It has to be pretty simple: touch and play. That means that every object can be selected, moved around and through interaction it can make some music.

And that is how everything started……

Four: constraints in the creative process

The whole concept described above may sound to you simple.

But every technology hides some constraints for the creative process. It is the same when a composer wants to write a piece of music for an orchestra. He knows that he wants to use an orchestra but he doesn’t know from the beginning if he is going to use a full orchestra or a limited and a smaller version of it.

Sometimes I am wondering how much the technological means limit me from doing exactly what I want to do. There are limitations and there are problems but from my point of you, for each problem there is a solution! You just have to find the right tool to do the job!

It is like a circulus in probando! You can’t really tell if the means that you are using limit your creative process or your creative process expands in another level because of these means. Definitely I can say that the means form my creativity and sometimes make me think in a different way than I would usually do. It is always a work in progress and a long path of learning, learning and again learning.

Five: what I do, is there to be shared

The biggest motivation to this project, is to present art in another context and to reach out an audience who may not be familiar with art.

Of course what I am presenting is my artistic interpretation, my artistic perspective based on my imagination, my creativity and aesthetics but I’m sure that the project will wake up your curiosity and you will want to explore more!!