Mixing Painting and Performance

Style transfer algorithms harmonize two art expressions that were not meant to meet.

This animated video blends pictorial and performative art.

To dialogue, the artist duo Tina&Charly either paint or dance.
While motion is a fundamental aspect of their artistic approach, their paintings are simply not moving. Conversely, their video record all their dances but do not retain their painting aesthetic.

Tina&Charly paintings. On the left a dyptich “Feu Sacré” et “Cycle de l’eau”. On the right “Scroll” and “Swip” in the series “Toucher” that we did together, more details in this paper.

To fill the gap, we exploit algorithms to automatically stylized each frame of their videos with their paintings. We use neural style transfer algorithms.

Tina&Charly designed a performance where they dance on a beach on a windy day. This merges three dynamics. There is the steady recklessness of the waves, the gallop of the clouds in the sky and Tina&Charly's peaceful harmonic motions.

From left to right: preparatory drawing of Tina&Charly; a frame of their performance; the output of the style transfer algorithm STROTSS. We use STROTSS in particular (see our code) because of its copy past-effect. Since it is optimization-based, we had to process each frame one after the other.

In the stylized frames, the different dynamics create moving sands in the foot of the dancers. The algorithm does not dissociate the sea from the sand so that the motions of the waves can spread to the sand, mixing the two elements.

Style transfer algorithms harmonize two art expressions that otherwise do not meet. The algorithm paints out video performances and animates still paintings.

Doing that manually, frame by frame, is a tedious and repetitive task, not much fulfilling for an artist. Style transfer algorithms offer a “simple” solution. However, could we proceed in a different way to animate these paintings and picturize the performance?

To answer, we thought of some paintings of cavemen. In the South of France, there are many impressive prehistorical caves, see the images below.

The lions and the horses in the Grotte Chauvet.

These paintings were made 30000 years ago or so, but the motion is ubiquitous. Can you imagine the lions rushing to their preys? The herd of bulls and horses circling around you? Or do you hear the muffled sound of the oversized rhinoceros crashing one into the other?

Actually, several prehistorians¹ think that the topography of the rock was also used to animate the overlapping drawings. These paintings were made for rituals.
They were probably designed so that the complex projection of our ancestors’ shadows, the darkness and the lack of oxygen would finally animate all the surroundings animals, uniting all the different artistic media.
Our ancestors lacked any type of material: no canvas or color tubes, and obviously not all the complexity of our computer and algorithms.
Still, we can imagine how captivating and fascinating these performances were.

Caveman paintings are hence a testimony that technology may be useful but is not necessary to animate paintings. Style transfer algorithms have helped us in a repetitive task. Yet, it is also depriving the artists and the spectators of imagining a step ahead from the still image.

Tina&Charly is a duo of French painters and performative artists. They are our friends, and we like to work with them to design machine and algorithms for creative processes — see, for instance, our other work here or this gallery. We use technology to give a pictorial sense of what it is — at the era of the algorithms — for artists to interact with machines. Hopefully, then, we will collectively cast a better vote for the technological environment we want.

[1]: Gwenn Rigal. (2016) “Le temps sacré des cavernes”. Les éditions José Corti.