Repository navigation
jump method is getting stuck #113
Description
I want to process a video frame by frame, but the jump(float where) method is either just taking ages or stuck in some kind of infinite loop or something like that. I used the Frames.pde example and just tweaked it a little for my needs.
Another problem is that mov.frameRate is -1 most of the time. I think both problems are caused by the fact that the movie just started playing very short before, but I might be wrong.
Am I doing something wrong or is this a bug?
I just changed very little:
/**
* Frames
* by Andres Colubri.
*
* Moves through the video one frame at the time by using the
* arrow keys. It estimates the frame counts using the framerate
* of the movie file, so it might not be exact in some cases.
*/
import processing.video.*;
Movie mov;
int newFrame = 0;
float framerate = 30;
void setup() {
size(1920, 1080);
background(0);
// Load and set the video to play. Setting the video
// in play mode is needed so at least one frame is read
// and we can get duration, size and other information from
// the video stream.
mov = new Movie(this, "video.mp4");
// Pausing the video at the first frame.
setFrame(0);
}
void movieEvent(Movie m) {
m.read();
}
void draw() {
background(255);
image(mov, 0, 0, width, height);
saveFrame("frames/frame" + nf(getFrame(), 3) + ".png");
fill(127);
text(getFrame() + " / " + (getLength() - 1), 10, 30);
newFrame++;
setFrame(newFrame);
}
int getFrame() {
return ceil(mov.time() * 30) - 1;
}
void setFrame(int n) {
mov.play();
// The duration of a single frame:
float frameDuration = 1.0 / framerate;
// We move to the middle of the frame by adding 0.5:
float where = (n + 0.5) * frameDuration;
// Taking into account border effects:
float diff = mov.duration() - where;
if (diff < 0) {
where += diff - 0.25 * frameDuration;
}
// The Program runs until here
mov.jump(where);
// But never executes the following code
mov.pause();
}
int getLength() {
return int(mov.duration() * framerate);
}
Activity
codeanticode commentedon Jul 17, 2019
@MCPepsh Try the latest beta, it fixes errors in the implementation of the time() and duration() functions that may be related to this issue.
neilcsmith-net commentedon Jul 19, 2019
@codeanticode there are multiple issues with the current implementation of
jump(), and the example above, that were discussed with Gohai a while back.Mainly, I highly recommend moving all GStreamer calls off the rendering thread and on to the GStreamer event thread using
Gst.invokeLater()or the executor. CallinggetState()off of the event thread, and definitely on the renderer thread, is problematic. You really need a way inside draw() to wait for a frame to be received to make the above code work.The AppSink.NEW_PREROLL implementation needs fleshing out - then the calls to play() and pause() in that example can be removed.
codeanticode commentedon Jul 19, 2019
@neilcsmith-net thanks for the comments. Could you point to some code demonstrating the correct use of Gst.Invokelater(). PRs are also appreciated :-)
neilcsmith-net commentedon Jul 19, 2019
Feel free to adapt anything from https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java It's under the same license.
async()is just a shorthand wrapper aroundGst,invokeLater()(or the executor, which amounts to the same thing).In particular, equivalent of jump() is at https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java#L171 and in handleSeek() - same seek code used in a few places.
neilcsmith-net commentedon Jul 19, 2019
And also https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/PImageSink.java which abstracts out the AppSink callbacks so they can be used in capture and video player.
codeanticode commentedon Jul 22, 2019
@neilcsmith-net could you expand on "The AppSink.NEW_PREROLL implementation needs fleshing out - then the calls to play() and pause() in that example can be removed." Thank you!
neilcsmith-net commentedon Jul 22, 2019
@codeanticode seeking (jump) while in paused state should work if you update and link in AppSink.NEWROLL to do basically the same as the new sample callback. That way, you don't have to do the quick play and pause again, which may cause problems and leave the seek in the wrong place anyway. Possibly better not to use the accurate seek flag here, or at least make it optional, too.
codeanticode commentedon Aug 1, 2019
Implemented with a95fcb7
codeanticode commentedon Aug 25, 2019
Beta 4 puts the seek operations in jump() in the GStreamer event thread, but not the play/stop/pause calls. All examples seem to work fine on Mac, but Scratch, which makes heavy use of jump and pause, sometimes (not always) hangs on Windows and Linux. Reopening, will revise in preparation for the v2.0 stable release.
hamoid commentedon Aug 25, 2019
I tried in Ubuntu 19.04. The examples
scratch,reverseandframesdon't seem to work properly.scratchgets stuck after a second or two.reversegives me a black screen.framesis initially black. Nothing seems to happen by pressing RIGHT a few times. If I hold it down, the video eventually shows up. But the frame counter is a bit wonky. It often shows -1. Sometimes seeking is very slow, so I see the counter change but not the frames. This is not due to the frames being repeated as they are, but I seek 10 or 20 frames forward and the image doesn't change.For this third example, what about having different background and text color? (both are 0). Also, what about reading
mov.frameRateandmov.duration()once inside setup? In getFrame() the frameRate seems to be hard coded to 30.In the console I see
Processing video library using GStreamer 1.15.90neilcsmith-net commentedon Aug 25, 2019
If you're going to do this, you'll need to move play/stop/pause into the GStreamer event thread, or in that example you'll end up with a thread race where you're probably mostly doing play/pause/jump
Maybe also take the play/pause/stop implementations from https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java That class is well battle tested for this particular use!
You need to fully implement the New Preroll listener to extract frames, which will mean no need to call play then jump then pause in the example at all - just jump in paused state. Those constant state changes are a possible issue in themselves.
Be a little wary of calling
getState(), at least on the Processing thread - maybe use the version with a timeout, and I'm not sure you need to in play / pause at all? Mind you, I am where the rate isn't 1, although it is in the GStreamer thread.Consider a short time out
getState()in seek, making sure it's in the GStreamer event thread (like at https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java#L285 ) - it's a little protection against swamping GStreamer with too many seek events while it's in the middle of a seek.Is that accurate or a bug in our version reporting? It's a non-stable GStreamer release.
codeanticode commentedon Aug 26, 2019
Cool thanks, I will look into Praxis' thread handling and see how I can adopt it into the video library.
As for the version reporting, the library is using Gst.getVersion()
codeanticode commentedon Jan 16, 2023
@hamoid do you still get playback issues using the latest version of the video library (2.2.x) on Linux?
hamoid commentedon Jan 17, 2023
Hi, I do still get issues. I just downloaded version 2.2.2 and placed it into the libraries folder. After restarting Processing 4.1.1 the examples are shown under contributed libraries (not as a built in library as before).
This is with an i9 cpu, mobile 3080 gpu, up-to-date Manjaro (Arch Linux variant).