RE: Excessive Frame Drops in pimg application during video capture by ABatWork
ABatWork
Fri Aug 08 09:32:01 PDT 2008
Hi Gary,
Did you get any feedback from pimg application owners regarding replacing ms
video renderer.
Also I tried some experiments to verify working of RGB overlays inside our
display driver. Also display drivers passes MSFT CETK, also default MSFT
provided mosquito application works on display driver, which actually creates
a RGB565 overlay surface.
Still we are getting poor performance because of preview. Because of same
some camera tests are failing. Also camera performance tests clearly says
encoding thread are starving beacause high of high cpu usage from preview
side.
Now Its really important for us to know, whether our MS video renderer is
using display driver provided overlay or not. Is there any way we can verify
this ?
Thanks
"fdbck" wrote:
> It does not appear that there is a registry key to change pimg to use
> something other than the DirectShow video renderer. I wasn't able to find one
> looking through the application code, but I haven't heard back from the
> application owners.
>
> The error you're seeing has an ASCII/unicode bug, "OK" and "wrong" are ASCII
> strings but it's a unicode debug output. It should say (input OK output
> wrong). I suspect this error is because the filtergraph mapper is trying to
> insert your filter in the middle between the vcap and the renderer, but your
> filter doesn't have an output since it is a renderer. That's just a guess
> based on the debug output and the fact that the video renderer is explicitly
> put into the graph.
>
> If you support RGB overlays in DDraw, the video renderer should dynamically
> switch over at run time. RGB overlays are ideal because the video renderer
> can switch between GDI and DDraw seamlessly without renegotiating the format.
> DDraw requires top down bitmaps, so make sure your biHeight is negative. If
> the height is positive then it'll never switch over to DDraw. YV12 is
> supported in the color converter, so i don't know why the RenderStream is
> failing. There must be something incompatible in the media type.
>
> The exception in pimg is probably the watson dump that the application team
> set up to occur any time there's a problem building the graph. The
> application throws an unhandled exception to trigger watson. Once you get the
> RenderStream to work the exception will disappear.
>
> Gary Daniels
> Windows Mobile Multimedia Test
>
>
> "ABatWork" wrote:
>
> > Hi Gary,
> >
> > We already have our video renderer, so writing a coustomized intermediate
> > filter won't be a good idea at this point. Also I am able to get more than
> > double performance with our renderer via our Dshow application.
> >
> > Also Can you please provide me any update regarding whether pimg
> > application have any registry key to change its default MS renderer to any
> > coutomized video renderer.
> > Infact We tried even connecting renderer, with our major GUID in camera
> > driver for its preview output format instead of MediaType_Video, and we got
> > following debug information from quartz prints:
> >
> > Types match: Req (2cfc3df4 e436eb7b) Found(2cfc3df4 e436eb7b
> > RegEnumFilterInfo: AB Video Renderer not wanted (input O output wog)
> >
> >
> > Where "AB Video Renderer" is our video renderer name. Can you please comment
> > on what is the meaning of ""(input O output wog)"".
> >
> >
> > Apart from above question, I am passing RGB data to preview pin from my
> > camera driver. To switch MS video renderer to Ddraw for using hw overlay,
> > should i pass YV12 data from our camera driver's preview pin.
> > Infact I tried even passing YV12 data from camera driver to its preview
> > pin. But got a exception in pimg application. While my Dshow based
> > application failed in RenderStream function. What can be possible cause for
> > this issue.
> >
> > Thanks in advance.
> > Ab
> >
> > "fdbck" wrote:
> >
> > > I'm not sure whether or not the pimg application has a registry key to
> > > control what video renderer it uses. I'll have to follow up on that.
> > >
> > > Ideally you would use the DirectShow provided video renderer rather than
> > > making a new one. If your overlay is available through DirectDraw then the
> > > video renderer will try to use it. There is one caveat with that, the video
> > > renderer requires an RGB fallback in the event the overlay in use cannot do
> > > color keying to deal with complex clipping surfaces... To do this the VR
> > > always negotiates an RGB format at graph build time, but then dynamically
> > > switches over to DDraw at run time. To get your custom format to negotiate to
> > > the VR you'll need to write a custom filter that can take in your custom
> > > format and convert it to RGB. When the dynamic format changes at run time the
> > > custom filter can change into a pass through filter with no impact on
> > > performance. It might sound like a lot of overhead, but it's probably easier
> > > than writing a video renderer. For the standard YUV formats the color
> > > converter can already handle them and do the pass through switch to have the
> > > best perf automatically. If your custom format could be adjusted to be YV12,
> > > YV16, UYVY, YVYU, YUY2, YVU9, Y41P, or CLJR then you could use the color
> > > convert to do this. We're working on fixing this requirement in the future.
> > >
> > > For the ptype of RenderStream, that's for the capture graph builder to
> > > identify the stream and not to specify the color format to render with.
> > > Assuming your video capture filter has only 1 preview pin, then
> > > RenderStream(&PIN_CATEGORY_PREVIEW, NULL, pVideoCaptureFilter, NULL,
> > > pVidRenderer) is functionally identical to what you're doing. I say this just
> > > in case you believed setting the pType could change the color format. To set
> > > the color format of the preview output pin use IAMStreamConfig.
> > >
> > > It sounds as though pVidRenderer points to your custom video enderer filter?
> > > It sounds like your filter is rejecting something with the video format
> > > coming from the video capture filter.
> > >
> > > If you're interested, there's some other spots you might be able to optimize
> > > perf. If you can have your camera driver use client unlimited buffering
> > > rather than driver limited buffering, then the DirectShow buffering filter
> > > will buffer the images from the driver without doing a memcopy and new
> > > buffers will be allocated and given to the driver whenever needed. If your
> > > hardware can do this it's a great thing to do, if your hardware cannot then
> > > the buffering filter will do the memcopy instead of your driver. If you have
> > > a high resolution still output and hardware jpeg encoding there's two
> > > options, you can encode to jpeg inside of your driver or you can expose the
> > > hardware through the GDI+ imaging library. If you can do both JPEG encode and
> > > decode in hardware through the GDI+ imaging library then there's some great
> > > potential for still image perf improvements.
> > >
> > > I'll get back to you on whether or not pimg can use your renderer.
> > >
> > > Gary Daniels
> > > Windows Mobile Multimedia Test
> > >
> > > "ABatWork" wrote:
> > >
> > > > Hi Gary,
> > > >
> > > > Thanks for awesome information. It proven to be very useful. And yes It was
> > > > preview only which was causing problem.
> > > > I tried video capture using Dshow based application which render only
> > > > capture stream and not preview, and 6 FPS performance for high resolution
> > > > changed to 30 FPS.
> > > >
> > > > Now to achieve performance, we left with only one option to change pimg
> > > > video renderer to our hw overlay renderer. First I tried first to implement
> > > > this our Dhsow based application which uses MS Video Capture filter.
> > > >
> > > > My camera driver supports Customized Major GUID(Lets name it as
> > > > MediaType_ABVideo). Same is supported inside our renderer. Here is the
> > > > function where I am getting failure:
> > > >
> > > > RenderStream( &PIN_CATEGORY_PREVIEW,
> > > > &MEDIATYPE_AbVideo,
> > > > pVideoCaptureFilter,
> > > > NULL, pVidRenderer);
> > > >
> > > > Where pVidRender is our Video Renderer.
> > > >
> > > > I turned on messages of quartz.dll and found following just before render
> > > > stream:
> > > >
> > > > EnumRegFilters returning 0 filters
> > > > ConnectVia: failed level 1
> > > > Connect Ended hr=80040217
> > > >
> > > > Which means VFW_E_CANNOT_CONNECT "No combination of intermediate filters
> > > > could be found to make the connection"
> > > >
> > > > Can you please comment your views on this.
> > > >
> > > > Also Later on If I try to use pimg application with our renderer, Will it be
> > > > possible?
> > > >
> > > > Thanks
> > > > Amit
> > > >
> > > >
> > > >
> > > > We have coustomized major type
> > > > "fdbck" wrote:
> > > >
> > > > > I suspect it's the buffering filter that's dropping rather than the vcap
> > > > > filter. The vcap filter doesn't drop frames, that's handled by the camera
> > > > > driver and the buffering filter.
> > > > >
> > > > > What has happened in the past is the CPU is unable to encode video fast
> > > > > enough to keep up with the frame rate that it's recieving the video, so the
> > > > > buffering filter buffers up the unencoded video until it runs out of ram.
> > > > > Once RAM is full the filter dropps the oldest frame in it's queue that hasn't
> > > > > been encoded and replaces it with the new sample it received. There's no
> > > > > logic to evenly distribute the dropped frames throughout the video file. Once
> > > > > the video capture is stopped the buffered video is encoded, which results in
> > > > > a delay from the time when you stop the capture to when the preview resumes
> > > > > and you're able to start the next capture. This algorithm works well for
> > > > > transient conditions, but not sustained problems with encoding. In extreme
> > > > > cases the resulting video file will look like a slide show with video frames
> > > > > only changing every few seconds until just before the end of the video, at
> > > > > which point the video will run smoothly for however much video was buffered
> > > > > up at the time the capture was stopped.
> > > > >
> > > > > If this is the behavior you're seeing, there's a few things that can be done
> > > > > to improve the quality of the video. Usually the problem is because the
> > > > > preview is using 100% of the CPU and leaving nothing for encoding video. The
> > > > > preview is using CPU because there may not be overlay hardware in the
> > > > > graphics chip, so it's rendered in software. The biggest thing you can do is
> > > > > if you have display hardware figure out what you need to do with the preview
> > > > > formats exposed by the camera driver to use hardware. Reduce the resolution
> > > > > of the preview, fewer pixels requires less CPU to move. Along with reducing
> > > > > the resolution of the preview, try to set a preview format that doesn't
> > > > > require any stretching or shrinking to be rendered. Unless the stretching or
> > > > > shrinking is done is hardware, it's extrememly CPU intensive. Reduce the
> > > > > frame rate of the preview, slower preview reduces CPU utilization. Once you
> > > > > look at how the preview window is rendererd and how much CPU it's using, try
> > > > > using the monte carlo profiling tools to see where the CPU is going.
> > > > >
> > > > > Gary Daniels
> > > > > Windows Mobile Multimedia Test
> > > > >
> > > > > "ABatWork" wrote:
> > > > >
> > > > > > I am working on ARMV4I device with WM6 OS using pimg application. My stream
> > > > > > interface camera driver followes 3 pin architecture and use "client limited"
> > > > > > model drived from MSFT provided NULLCAM driver.
> > > > > > For around 4 mins of wmv video capture i am getting around 30KB video
> > > > > > seems to have only 9-10 frames. I am passing RGB data on both Preview and
> > > > > > capture pin. Capture resolution is QCIF(176x144).
> > > > > >
> > > > > > Is there any way, by which i can tell Video Capture filter to not drop even
> > > > > > a single frames anyhow ?
> > > > > > Because of frames drops of video capture filter, we are getting very bad
> > > > > > quality and performance. Is there any way we can overcome this problem?
> > > > > >
> > > > > > Thanks
> > > > > > ab