How it works
Benjamin Mock mail [at] benjaminmock.de
As an avid podcast listener, I oftentimes heard great passages that I wanted to share somehow. In a two hour podcast, there's maybe just this one sentence that grabs my attention, that I want to share with others. Of course, I could point them to the podcast and tell them to listen at 1h23:47, but who does that?
So I was looking for a tool to extract this highlight and turn it into a video, so I could share it on social media because that's not possible with audio-only content. But I could not find the right one for me, so I decided to build clips.fm as a proof of concept, to see if it's possible to build such a tool completely in the browser. Of course, it wasn't called clips.fm back then. It all started as a jsfiddle with more than 250 versions, where I tried to merge the audio stream of an mp3 with the stream of a canvas. After that worked I built the first version of the audio visualization.
There were a lot of technical challenges to overcome on the way:
- displaying the waveform of an mp3 for snippet selection
- drawing a waveform animation on the canvas capturing the audio
- stream and the canvas animation and merging it creating a
- downloadable video of the stream
For some I found great libraries like wavesurfer to display the waveforms for the snippet selection. For others, like animating the waveform within the canvas I had to write my solutions.
And then there are really hard problems like video formats. Linkedin for example is very flexible when it comes to video formats. The generated webm works there directly. That's cool. But Twitter doesn't like it at all, it's very restrictive when it comes to the format. So I had to start transcoding the video using the wasm version of ffmpeg, which also runs in the browser.
And I even discovered a bug in Chrome when generating the video. Because of this bug, the video is not searchable after it was generated. That means you have to watch from the beginning, you can not jump to a specific position. I fix this by transcoding it before downloading.