Intro to ffmpeg: audio-to-video filters

Thanks to the interest of Oddlogic and BFNK in creating an automated way to create visual content based on our music, I took their advice on looking up to ffmpeg. Ffmpeg is "A complete, cross-platform solution to record, convert and stream audio and video", free & open-source. Ffmpeg makes easy and fast to do all kind of manipulations to video and audio. Installation steps are here.

ffmpeg is the audio/video converter. It requires always an input and an output. ffplay is the player included. It only requires an input.

For ffmpeg the syntax is:

1ffmpeg -f lavfi -i INPUT -vf filtername OUTPUT

For ffplay the syntax is:

1ffplay -f lavfi movie=input.mp4

We will focus on the audio-to-video capabilities.

When creating video from audio, we need to duplicate the input so we can add it to the resulting video. A nice example from the dev site is:

1                [main]
2input --> split ---------------------> overlay --> output
3                |                                            ^
4                |  [tmp]                        [flip]  |
5               +-----> crop --> vflip -------+

1ffmpeg -i INPUT -vf "split [main][tmp]; [tmp] crop=iw:ih/2:0:0, vflip [flip]; [main][flip] overlay=0:H/2" OUTPUT

So basically you have to split your media and then put them together after processing them. That's all, but it's important to remember when applying more than 1 filter.

A list of audio-to-video filters:

• ahistogram

• aphasemeter

• avectorscope

• showcqt

• showfreqs

• showspectrum

• showwaves

• showvolume
  
  The one that have the weirdest results imo is avectorscope. But the magic is in the combination of several filters in serie or parallel. For example,

1ffplay -f lavfi 'amovie=input1.wav, asplit [a][out1];[a]avectorscope=m=polar:mirror=x:draw=line:s=600x400:r=60:zoom=5:rc=165:gc=82:bc=0:rf=5:gf=10:bf=15'

Adding now a filter at the end (mirror effect),

1ffplay -f lavfi 'amovie=input.wav, asplit [a][out1];
2[a]avectorscope=m=polar:mirror=x:draw=line:s=600x400:r=60:zoom=10:rc=165:gc=82:bc=0:rf=5:gf=10:bf=15' -vf "crop=iw/2:ih:0:0,split[left][tmp];[tmp]hflip[right];[left][right] hstack"

And that is most of it! now it is only a matter of testing new values, filters, etc. Some of the results I've been getting are:

My main goal here is not to become some serious programmer/databending wizard. I'm not a visual artist and my knowledge and experience in this topic is almost null. But with tools like this I found very exciting and easy to have a visual representation of your own music that you can keep processing and processing for much better results.

And I think you can do that too!

Next step would be to start creating some shell scripts to go further in manipulating the parameters.
If you are curious and want to do more, check "frei0r" plugins for cool video effects.

Links: ffmpeg website: https://ffmpeg.org
Filters documentation: https://ffmpeg.org/ffmpeg-filters.html
Example of filters: https://trac.ffmpeg.org/wiki/FancyFilteringExamples