[gephex-devel] More questions

[Martin Bayer]

> There is one
> important aspect of the software we are developping that I'd like to 
> exchange with you about : frame rates. By that I mean update frequencies 
> of the units (what we call gears).   ...

> - what determines the frame rate shown as output ? If your using an mpeg 
> movie as input, then the output should probably be synched with the 
> source. But if your source is generated live, i.e let's say opengl, than 
> what sets the frame rate of the output ?

In the 0.4.x branch there is just one global update frequency for all
data sinks in the data flow graph. This frequency is a fixed parameter
and doesn't change while processing. There is no difference in the
handling of video, audio, numeric and custom data links.

A video file with a different frame rate is just (sub/super)sampled at
the needed positions. This is only possible for random access video
sources. Video sources with a stream behavior (webcam, mpeg stream)
this is not possible. The current implementation doesn't solve these
cases in a clean way. For a webcam like video source a solution is to 
duplicate the last image if there are no pending new images in the input 
buffer. But this leads to a lot of unneeded recalculation.

> -One goal for drone was to process video and audio as well. But while 
> video usually has a frame rate of  ~25fps, using the same update 
> frequency for audio processing would induce a ~40ms latency : the 
> latency between the moment the sound is captured and the time it is 
> outputed after processing. That can be too much to cope with,  for 
> example when doing live performances. So i'm wondering how Gephex 
> handles this issue. Does it handle audio differently from video ?

The main reason for the introduction of audio streams was
signal analysis like detection of beat and frequency distribution for
influencing the video generation process with the extracted data. A
latency at or below the video frame rate is not problem for that usage
scenario.

The audiooutput and audiosynth code was more a proof of concept and is
in my opinion not really useful at the moment. The Linux audio
processing tools are very advanced and well integrated. One way to bring
a usable audio latency to the video world could be a to communicate
better with that world. E.g. a common timebase for jackd and the video
applications could be a solution.

> For example, audio units could be run in a separate thread at a higher 
> rate, while video units would be run in another one at a slower rate. 
> The problem with that approach is that units having both types for input 
> have to be treated in a special way : i.e. having 2 different functions.

What about a chain like

audio -> number -> video -> number -> audio

You need to update the video link at the audio rate to reduce the
latency to audio level.

Processing signals with different sample rates is not trivial. Even in a 
video only environment there are still these problems left.

- mixing two videosources with 24 and 25 fps
- video sources with variable framerates (webcams)
- outputs with different framerates
...

One plan to eliminate some of these problems is to attach a "valid"
timeinterval to each dataobject. If the current time is in the
interval the data objects doesn't need to be recalculated.

E.g. a video source with 25fps writes frames with valid intervals 
[t,t+1/25s[ and an audio source can use a valid interval of 
[t,t+buffsize/samplerate] for a audiobuffer of variable size.

The update algorithm now doesn't need any special knowledge about the 
data types. It just uses the valid intervals that are set by 
(intermediate) data sources. A module/unit/gear must calculate the valid 
interval of its outputs from the valid intervals of its inputs.


This timestamping thing is still a very raw draft and there are many 
issues to think about left:

- feedback/recursion
- possible deadlocks
- variable samplerates at sources and how to predict the time interval
- multiple updates if the timeintervall is still not valid after an update


Martin



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 256 bytes
Desc: OpenPGP digital signature
Url : http://lists.gephex.org/pipermail/gephex-devel/attachments/20041215/f8d5ff1e/signature.bin