For some reason, the larger the video to process, the slower FileVideoStream is?

Is that intended behavior or perhaps my issues is somewhere else.

Right now when processing frames a 20 second video will read each frame super fast, while a 4 minute video will take forever to read just one frame. What is goin on?

What kind of processing were you applying on the vid? Can you share code to reproduce your issue?