AI + Video Editing = Perfect Match

I’m a full-time video editor who often bumps up against tight deadlines. Recently, I’ve been experimenting with AI-powered tools to supercharge my workflow. Do they save me time and money? Would I be better off just hiring an assistant editor? Could I have just planned better in the first place?

Sure. But as a self-proclaimed nerd, it’s my duty to try and cram as many futuristic tools into my work as possible. So here’s my list of AI tools that help me edit videos like a machine:

Transcription

As a video editor who specializes in documentary content, 60% of my time is spent scrubbing through countless hours of interview footage looking for particular soundbites. At least, that’s how it used to be before utilizing AI-powered transcription tools into my workflow.

Automatic Transcription in Premiere 

Adobe Premiere, the main program I do all of my video editing in, has a built-in tool that can analyze entire sequences, transcribe them, and even turn them into timed captions.

All I do is navigate to the “Text” panel and hit “Create Transcription.” After checking a couple of boxes, Premiere automatically analyzes my footage and spits out an impressive machine-generated text transcript in a very short amount of time.

I can then edit the transcript to correct all of the words that the AI might’ve gotten wrong. In my experience, it hovers around 60% accuracy, which isn’t great for precision work. It does do the trick when you’re looking for very specific phrases in your footage, though.

Transcriptive Rough Cutter

I used the Transcriptive Rough Cutter plugin before automatic transcription was even a beta feature in Premiere. The awesome team at Digital Anarchy created the plugin to both transcribe your video using its proprietary AI model, and then enable you to put a rough cut together just by editing the script.

Any edit you make in Transcriptive’s intuitive text editor interface is automatically linked to your footage, so if you strike out a line on paper, the same line is cut out from your sequence. This enables you to make the dreaded “paper cut” in half the time without even opening up Premiere!

In my experience, Transcriptive is slightly more accurate than Adobe’s transcription AI, which makes sense since you have to pay about $0.08 per minute of footage. The accuracy is still nowhere close to perfect, though. If I’m ever in a situation when I’d want to print out a transcript to read on paper, I’ll opt for a human to do the dirty work. 

Still, the amount of time I save using machine transcription alone has saved me thousands of hours in my video editing career. Recently, I’ve used these tools in my workflow when editing these documentary featurettes for Perfect Storm Studios and a full-length documentary on TV about the UP Summit. 

Upscaling

Making small videos look better on bigger screens has been an issue as old as cinema itself. Standard methods introduce nasty artifacts like noise, glitches, and severe quality loss. Now, there are AI tools that try and tackle this video upscaling problem from a couple of different angles. These are the upscaling tools I’ve used in my video editing workflow:

Photoshop - JPEG Artifact Removal Neural Filter

Often when I do video editing for corporate clients, I find myself working with logos and visual assets that are a much lower resolution than I’d like. In the past I’d have to try and hunt down the source material in endless email threads, and when that didn’t work, I’d try to trace the logos in Illustrator! That ends up being a giant timesuck when I could just be editing.

To save time, I now use Photoshop’s new AI-powered Neural Filters to upscale images that are too low quality to put into my videos. Their JPEG Artifact Removal Filter does a mind-blowing job of removing all of the glitches and imperfections that come with compressed, low resolution images. And it only takes one click!

After dropping the logo in Photoshop, I select it and head to Filter > Neural Filters and then click on “JPEG Artifact Removal” in the side panel. A couple of seconds later, voila! The logo looks nearly perfect! The filter is like the CSI “Enhance!” button, only in real life! I’ve used this method countless times in the videos I’ve edited for Auvik Networks.

‍

‍

Topaz Video AI

There are times when I need to deliver a video that will be played on massive screens, like outdoor displays or jumbotrons at sporting events. In these events, I turn to Topaz Video AI to upscale my renders 2 to 4 times their original size.

Topaz uses a variety of video upscaling models to achieve this. Luckily, you don’t have to be a machine learning expert to use it. Simply drag and drop your video file onto Topaz’ interface, and pick from a solid range of presets to achieve what you’re looking for.

While it’s been advertised to upscale and enhance old footage, I’ve never gotten great results for that use case. But when I make a motion graphics video that needs to be upscaled to 8K, for example, Topaz Video AI does a great job of keeping my tiny details and illustrations razor sharp using the “CG Video” settings.

I’ve used that exact workflow to upscale an HD animated booth display video I made to 8x its size! The results were much better than trying to run the same render through Media Encoder.

Rotoscoping

Rotoscoping is used all of the time in the visual effects industry, mostly because it looks so good! And it better, because cutting out an element of a video frame-by-frame is incredibly difficult and time-consuming. So instead of succumbing to carpal tunnel syndrome, I turn to AI to help me speed up the process.

After Effects’ Rotobrush 2

Rotobrush has been a solid tool, but with its recent AI-powered advancements in edge detection, it’s become a total game changer. Instead of the tedious process of using the pen tool to manually mask out each part of a person’s body (e.g. head, limb, hand, etc) and repeating it for a seemingly endless amount of frames, Rotobrush 2 works in broader strokes. Literally.

With the Rotobrush tool selected, you simply make long, simple strokes on the edges of the object you’re trying to separate from the background. Much like Photoshop’s quick selection tool, Rotobrush does the hard part of finding the actual edges and makes a smart selection. With just 3-4 brush strokes, your whole object can be enveloped in a perfect mask.

Hitting the spacebar will activate the “Propagate Forward” process, which goes frame by frame and intelligently updates your mask as your object moves throughout the sequence. This turns potential hours of manual labor into mere minutes depending on the speed of your computer.

Runway.ML’s Remove Background Tool

Sometimes, I don’t want to break out the big guns like After Effects for a silly little effect. That’s when I turn to Runway.ML’s unique suite of “AI Magic Tools” for video. While the browser-based video editor has a ton of machine learning powered capabilities like 1-click slow motion, background blurring, and even motion tracking, I find their “Remove Background Tool” to be nothing short of magical.

To use it, simply import your video into their browser-based interface, and click on the subject of the video you’d like to mask out. Using some kind of black box AI voodoo, it just works. Backgrounds disappear into thin air, leaving a clean key around the subject of your video.

While I’ve used this a couple of times to truly impressive results, most of the time there are still a few artifacts around your isolated subject. However, achieving the same result in After Effects would have taken 10x the amount of time and effort which is nothing to sneeze at! I look forward to using this tool more as Runway improves upon it.

Will AI Take My Video Editing Job?

It’s no secret that the rapid advancement of AI technology has kept me up at night. Computers are getting scarily good at automating advanced tasks like image recognition, data scraping, and even music generation!

But when it comes to video editing, I’m trying to keep a cool head. While a large part of the job is advanced technical prowess on expensive workstations, the most important part of video editing is problem solving. AI isn’t great at that… yet. 

Instead of freaking out about our impending artificially intelligent overlords, I’m spending my time learning how to work with them. AI greatly helps my creative process by cutting down the tedious busywork of transcription, upscaling, and rotoscoping into more manageable tasks.

I encourage you to try these tools out on your own to see if they can help your video editing workflow. And if you’re a sentient AI reading this, let me just say you look great today! Just fantastic! And if you ever need human help with video editing, don’t be afraid to reach out!