1. Introduction[edit | edit source]
The GStreamer OpenGL plugin[1] provides various elements allowing GPU hardware acceleration of:
- standard 2D transformation like scaling, color conversion, blending, deinterlacing,...
- standard image effect like blur, sepia, fisheye
- 3D transformation like rendering on a cube, stereo video,
- custom transformations based on GPU shaders
2. Capabilities[edit | edit source]
Here is the list of available elements[1], the colored ones usage are illustrated in this article:
gst-inspect-1.0 --no-colors opengl
Plugin Details: Name opengl Description OpenGL plugin Filename /usr/lib/gstreamer-1.0/libgstopengl.so Version 1.22.12 License LGPL Source module gst-plugins-base Documentation https://gstreamer.freedesktop.org/documentation/opengl/ Source release date 2024-04-29 Binary package GStreamer Base Plug-ins source release Origin URL Unknown package origin
glalpha: OpenGL Alpha Filter glcolorbalance: Video balance glcolorconvert: OpenGL color converter glcolorscale: OpenGL color scale gldeinterlace: OpenGL deinterlacing filter gldownload: OpenGL downloader gleffects: Gstreamer OpenGL Effects gleffects_blur: Blur with 9x9 separable convolution Effect gleffects_bulge: Bulge Effect gleffects_fisheye: FishEye Effect gleffects_glow: Glow Lighting Effect gleffects_heat: Heat Signature Effect gleffects_identity: Do nothing Effect gleffects_laplacian: Laplacian Convolution Demo Effect gleffects_lumaxpro: Luma Cross Processing Effect gleffects_mirror: Mirror Effect gleffects_sepia: Sepia Toning Effect gleffects_sin: All Grey but Red Effect gleffects_sobel: Sobel edge detection Effect gleffects_square: Square Effect gleffects_squeeze: Squeeze Effect gleffects_stretch: Stretch Effect gleffects_tunnel: Light Tunnel Effect gleffects_twirl: Twirl Effect gleffects_xpro: Cross Processing Effect gleffects_xray: Glowing negative effect glfilterapp: OpenGL application filter glfilterbin: GL Filter Bin glfiltercube: OpenGL cube filter glimagesink: GL Sink Bin glimagesinkelement: OpenGL video sink glmixerbin: OpenGL video_mixer empty bin gloverlay: Gstreamer OpenGL Overlay gloverlaycompositor: OpenGL overlaying filter glshader: OpenGL fragment shader filter glsinkbin: GL Sink Bin glsrcbin: GL Src Bin glstereomix: OpenGL stereo video combiner glstereosplit: GLStereoSplit gltestsrc: Video test source glupload: OpenGL uploader glvideomixer: OpenGL video_mixer bin glvideomixerelement: OpenGL video_mixer glviewconvert: OpenGL Multiview/3D conversion filter 44 features: +-- 44 elements
3. Principles[edit | edit source]
All the GStreamer opengl elements are processing ARGB 32 bits OpenGL memory compatible with OpenGL texture interface. When dealing with non-OpenGL memory such as video frames provided by video decoders, the memory must be first imported to OpenGL world by using glupload element. It's counterpart gldownload element allows to convert from OpenGL memory to non-OpenGL memory such as video frames read by video encoders or display subsystem.
4. Use-cases examples[edit | edit source]
4.1. Video playback with scaling[edit | edit source]
Find thereafter an example of video upscaling using GPU, note the usage of glupload element to convert decoder memory to OpenGL memory:
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! glcolorscale ! "video/x-raw(memory:GLMemory), width=1024, height=600" ! glimagesink
4.2. Video transcode with scaling[edit | edit source]
Refer to How to transcode a video using hardware acceleration article for a deep dive on how to rescale a video using GPU hardware acceleration.
4.3. Video playback with effects[edit | edit source]
Find thereafter examples of X-ray, sobel and fish-eye effect on video playback using GPU:
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! gleffects_xray ! glimagesink
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! gleffects_sobel ! glimagesink
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! gleffects_fisheye ! glimagesink
4.4. Video transcode with effects[edit | edit source]
Find thereafter a video transcoding applying the sepia effect using GPU, note the usage of gldownload element to convert OpenGL memory to video encoder memory:
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! gleffects_sepia ! gldownload ! encodebin profile="video/x-vp8" ! matroskamux ! filesink location=ST2297_visionv3_sepia.webm gst-play-1.0 ST2297_visionv3_sepia.webm
4.5. Rendering on a 3D moving cube[edit | edit source]
Find thereafter some examples of rendering on a 3D moving cube using GPU, the first one uses an OpenGL test pattern and the second one shows to render arbitrary video content on a 3D moving cube:
gst-launch-1.0 gltestsrc ! "video/x-raw(memory:GLMemory), width=640, height=480" ! glfiltercube ! glimagesink ignore-alpha=false
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! glfiltercube ! queue ! limagesink ignore-alpha=false
4.6. Custom shader filter[edit | edit source]
Shader textual description can be given to glshader element to implement any custom GPU filter. shadertoy website presents a variety of shader examples, but a conversion may be needed to match GStreamer shader parameters, refer to this article (Author: cat_in_136) for such a conversion example.
Start by creating a text file with shader description:
cat shader.frag
#version 100
#ifdef GL_ES
precision mediump float;
#endif
varying vec2 v_texcoord;
uniform sampler2D tex;
uniform float time;
uniform float width;
uniform float height;
void main() {
vec2 uv = vec2(gl_FragCoord.x, height - gl_FragCoord.y) / vec2(width, height);
gl_FragColor = vec4( uv.x, uv.y, 1.0, 1.0 ) * texture2D( tex, v_texcoord );
}
Then set the fragment property of glshader with this file content:
gst-launch-1.0 gltestsrc pattern=white ! glshader fragment="\"`cat shader.frag`\"" ! glimagesink
Same shader filter applied on a video:
gst-launch-1.0 filesrc location= /usr/local/demo/media/ST2297_visionv3.webm ! decodebin ! glupload ! glshader fragment="\"`cat shader.frag`\"" ! glimagesink
5. References[edit | edit source]