Stylizing Video

It is a new way to generate video content by get style from one or more keyframe then automatically propagates the stylization to the rest of the sequence.

I was inspired by the "Stylizing Video by Example" project at SIGGRAPH 2019. The paper was publish in ACM Transactions on Graphics 38 (SIGGRAPH 2019, Los Angeles, USA, July 2019). you can download Stylizing Video by Example Paper (full)

From team's member list I think the project is co-operate between Czech Technical University and Adobe Research (Please correct me if incorrect).

Stylizing Video by Example's Abstract

"We introduce a new example-based approach to video stylization, with a focus on preserving the visual quality of the style, user controllability and applicability to arbitrary video. Our method gets as input one or more keyframes that the artist chooses to stylize with standard painting tools. It then automatically propagates the stylization to the rest of the sequence. To facilitate this while preserving visual quality, we developed a new type of guidance for state-of-art patch-based synthesis, that can be applied to any type of video content and does not require any additional information besides the video itself and a user-specified mask of the region to be stylized. We further show a temporal blending approach for interpolating style between keyframes that preserves texture coherence, contrast and high frequency details. We evaluate our method on various scenes from real production setting and provide a thorough comparison with prior art."

Stylizing process - How to do

I will prepare video then replace other face with some art style. I will use an image to generate new video. Let's see how it work.

Prepare video - I prepare a short video from PM. Prayut Chan-o-cha.

Prepare sequence images - It is extract as image or make frame-by-frame images from the video. I use ffmpeg command ffmpeg -i source.mov thumb%04d.jpg -hide_banner to generate the sequence images. Result is 179 photo files, amount of the images depend on video source duration.

Prepare image key for stylizing - I choose first image (0001.png) then replace a face on PM. Prayuth and add art style by online service name GoArt AI photo effects It prepare many style - for example Sketch, Van Gogh, Ukiyoe, Watercolor, Byzantine, Picaso, Rust, Pop Art.

Generate new content

• Sequence images - 179 images
• Stylizing image key - 1 image (please note more key = more quality)

So I will create video which automated propagates by stylizing image key to the rest of the sequence images. The paper provide demo tool name ebsynth. you can choose stylizing image key and sequence images then input amount of keyframe which you want.

Combine new sequence images to Video - I use command ffmpeg -i %06d.png -r 25 -c:v libx264 -crf 20 -pix_fmt yuv420p output.mp4 to combine sequence images back to video

That's it. let's see the result https://youtu.be/BpmYpj_6qhY