Render to Texture: Fixed Function vs. Modern OpenGL

Welcome back! Last time we talked about texturing, lighting and geometry in regards to porting fixed function OpenGL to Modern OpenGL and while it covered the basic features in OpenGL programs, it left out a lot of advanced techniques that your fixed-function code may have used.  In this article I’m going to talk about one of those advanced features, render-to-texture.

Render-to-Texture

Render to texture is used in a variety of graphical techniques including shadow mapping, multi-pass rendering and other advanced visual effects.  It involves taking the results of one rendering pass and using those produced pixels as a texture image in another rendering pass.  There a few ways you could have done this in your fixed-function applications: 

1. Render to back buffer, read the pixels back via glReadPixels, and then set the texture image using those pixels via glTexImage2d or glTexSubImage2d.    This isn’t very performant since the pixel data is moving from video card, back to CPU ram, and then back to the video card again.
2. Render to back buffer, copy the pixels to the texture image via glTexCopy2D.  This is better since it keeps the pixel data on the video card, but it is still making a copy.
3. Render to the texture image directly on the video card.  This is the best since no copies are involved.

In this article we’re going to talk about scenario number three exclusively as this is approach most likely taken by your existing code base.

PBuffers

Pixel buffer (pbuffers for short) are fixed-function off-screen memory areas on the video card that replace the normal back-buffer as the destination for rendering.   In normal rendering there is a font-buffer, a back-buffer, a depth buffer and (optionally) a stencil buffer allocated for you when you create the OpenGL graphics context associated with your window.  This default set of buffers is called the window-system-provided framebuffer. The front-buffer is the video card memory that’s actively being scanned out on your monitor and the back buffer is the target for pixel data that’s begin generated when you render with OpenGL draw calls.  What a pbuffer does is provide another set of buffers to use as a destination for the pixel data created when you render with OpenGL.  

Pbuffer support falls into the category of an OpenGL extension.  An OpenGL extension is an optional feature of OpenGL that some video card vendors can elect to support.   Like creating an OpenGL context, it is also a platform-specific OpenGL extension.  This means that there’s a different implementation with different API calls for each platform including Windows, Mac, Linux and a variety of embedded platforms like Android and iOS.  Here, I’m going to discuss the Windows version of pbuffers and its API described by the two OpenGL extensions WGL_ARB_pbuffer and WGL_ARB_pixel_format. 

With these two extensions you now can render an image to the pbuffer off-screen memory on your video card.  The question becomes; how can you use it in your next rendering pass as a texture image?  The answer is yet another OpenGL extension WGL_ARB_render_texture.  What this extension does is let you describe the type of texture image (pixel format and texture-target) that you want your off-screen pbuffer to be used for.  With these three extensions we have everything we need to create a copy-free fixed-function render-to-texture application.

The following procedure describes exactly how to use the APIs in these extensions to perform render-to-texture.  It is taken directly from the WGL_ARB_render_texture specification and paraphrased:

1. Create the rendering window. Call wglChoosePixelFormatARB and find a suitable pixel format for rendering the image. Set the pixel format for the rendering window to this pixel format.
2. Create a context for the window. Make the context current to the window and initialize the contexts attributes. Bind a texture object to the GL_TEXTURE_2D target and set the texture parameters to the desired values.
3. Create the pbuffer. Call wglChoosePixelFormatARB and find a suitable pixel format for rendering the texture. WGL_DRAW_TO_PBUFFER and (WGL_BIND_TO_TEXTURE_RGB_ARB or WGL_BIND_TO_TEXTURE_RGBA_ARB must be TRUE). Create the pbuffer with this pixel format. Set the pbuffer width and height to the width and height of the level zero image. Set WGL_TEXTURE_FORMAT_ARB to be WGL_TEXTURE_RGB_ARB or WGL_TEXTURE_RGBA_ARB. Also set WGL_TEXTURE_TARGET_ARB to WGL_TEXTURE_2D_ARB.
4. Create a context for the pbuffer. Make the pbuffer context current and initialize the context's attributes.
5. Render the image to the pbuffer. Call glFlush.
6. Make the window context current and call wglBindTexImageARB to bind the pbuffer drawable to the texture created in step 2. Set <iBuffer> to WGL_FRONT_LEFT or WGL_BACK_LEFT depending upon which color buffer was used for rendering the pbuffer.
7. Render to the window using the pbuffer attached to the 2D texture.
8. Call wglReleaseTexImageARB to release the color buffer of the Pbuffer from the texture. Goto step 5 to generate more frames. 

Wow, that looks like quite a few steps to follow, but really it isn’t that bad at all.   As a demonstration, let’s take our last fixed-function example (textured, lit, rotating cube) and use a pbuffer to create an interesting “hall of mirrors” effect.  What we’ll do is render the scene twice.  The first time we’ll render this last example’s scene into the pbuffer.  Then we’ll render that same scene again, but instead of using the old “circle” texture we’ll use the results from our first rendering pass.

Figure 1 shows the result Listing 6.1 shows the code that produced it.

FrameBufferObjects

Now that we’ve got copy-less render-to-texture going in our fixed-function example let’s move on to doing this with Modern OpenGL, but first, a thing to note with the previous example. When you use pbuffers you also must use separate rendering contexts (one for each pbuffer) and that brings along a host of issues since you now have to manage two (or more) completely separate OpenGL states.

Modern OpenGL solves this problem in an entirely different way with something called a FrameBufferObject or FBO for short.   A FBO is a complete description of a set of compatible (with each other) off-screen rendering surfaces, including up 8 (or more) color-surfaces, a depth buffer surface, and a stencil surface.  These various buffers are called framebuffer-attachable images and you’ll have to create and manage them yourself.  You pretty much get everything you had with pbuffers but with the big difference of (besides having multiple color surfaces) having it all done in the single OpenGL context associated with your window.  This means there is no context-switch overhead and no need to manage separate OpenGL states.

Let’s take a close look at this FBO thing and see what it’s about.   We already know that a FBO contains a set of framebuffer-attachable images, but exactly what are they?  There are off-screen memory buffers on the video card that can be used as the target and source of pixels produced and consumed when rendering in OpenGL.  There are two types of framebuffer-attachable images; texture images and renderbuffer images.  You use texture images when you want to use the resulting pixels as a texture in a later rendering pass and renderbuffer otherwise.

Each of these framebuffer-attachable images is assigned a specific attachment position in the containing framebuffer object.  There are attachment points for multiple color buffers (GL_COLOR_ATTACHMENT0 … GL_COLOR_ATTACHMENTn, where n can be queried using GL_MAX_COLOR_ATTACHMENTS) and either a depth buffer (GL_DEPTH_ATTACHMENT) and a stencil buffer (GL_STENCIL_ATTACHMENT), or a combined depth-stencil buffer (GL_DEPTH_STENCIL_ATTACHMENT).

Framebuffer objects are optimized for attaching/detaching the image buffers associated with these attachment points.  Doing this should be preferred over creating multiple FBOs, assigning a static set of image buffers and then switching between those FBOs.

You use glGenFramebuffers() and glDeleteFramebuffers() to create and destroy FBO objects.  You use glBindframeBuffer() to make that frame buffer the target for active rendering.   You use glGenRenderbuffers() to create render-buffer object, glBindRenderbuffer() to make it active, and glRenderbufferStorage() to actually create the off-screen memory buffer.  You use glFramebufferRenderbuffer() and glFrameBufferTexture2D() to attach previously created render-buffer or texture-images to a framebuffer object.

Once you’ve attached all the texture-images and render-buffers you want to use to the FBO you must call glCheckFramebufferStatus().  glCheckFrameBufferStatus() is used by the OpenGL device driver to validate the compatibility of the various buffers with each other and with the video card’s hardware.   If, and only if, the driver returns GL_FRAMEBUFFER_COMPLETE are you free to actually use the FBO.

When the driver doesn’t return GL_FRAMEBUFFER_COMPLETE is can be kind of tricky to figure out what the problem is.  The OpenGL spec defines a set of “completeness rules” that you must satisfy to create a valid FBO.  Here they are copied directly from https://www.opengl.org/wiki/Framebuffer_Object ¹ and edited for clarity.

Attachment Completeness

Each attachment point itself must be complete according to these rules. Empty attachments (attachments with no image attached) are complete by default. If an image is attached, it must adhere to the following rules:

• The source object for the image still exists and has the same type it was attached with.
• The image has a non-zero width and height (the height of a 1D image is assumed to be 1). The width/height must also be less than GL_MAX_FRAMEBUFFER_WIDTH andGL_MAX_FRAMEBUFFER_HEIGHT respectively.
• The layer for 3D or array textures attachments is less than the depth of the texture. It must also be less than GL_MAX_FRAMEBUFFER_.
• The number of samples must be less than GL_MAX_FRAMEBUFFER_.
• The image's format must match the attachment point's requirements, as defined above. Color-renderable formats for color attachments, etc.

Completeness Rules

These are the rules for framebuffer completeness. The order of these rules matters.

1. If the target​ of glCheckFramebufferStatus​ references the Default Framebuffer (ie: FBO object number 0 is bound), and the default framebuffer does not exist, then you will get GL_FRAMEBUFFER_UNDEFINED. If the default framebuffer exists, then you always get GL_FRAMEBUFFER_COMPLETE. The rest of the rules apply when an FBO is bound.
2. All attachments must be attachment complete. (GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT when false).
3. There must be at least one image attached to the FBO. The GL_FRAMEBUFFER_DEFAULT_WIDTH and GL_FRAMEBUFFER_DEFAULT_HEIGHT parameters of the framebuffer must both be non-zero. (GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT when false).
4. Each draw buffers must either specify color attachment points that have images attached or must be GL_NONE. (GL_FRAMEBUFFER_INCOMPLETE_DRAW_BUFFER when false). Note that this test is not performed if OpenGL 4.1 or ARB_ES2_compatibility is available.
5. If the read buffer is set, then it must specify an attachment point that has an image attached. (GL_FRAMEBUFFER_INCOMPLETE_READ_BUFFER when false). Note that this test is not performed if OpenGL 4.1 or ARB_ES2_compatibility is available.
6. All images must have the same number of multisample samples. (GL_FRAMEBUFFER_INCOMPLETE_MULTISAMPLE when false).
7. If a layered image is attached to one attachment, then all attachments must be layered attachments. The attached layers do not have to have the same number of layers, nor do the layers have to come from the same kind of texture (a cubemap color texture can be paired with an array depth texture) (GL_FRAMEBUFFER_INCOMPLETE_LAYER_TARGETS when false).

Notice that there is no restriction based on size. The effective size of the FBO is the intersection of all of the sizes of the bound images (ie: the smallest in each dimension).

These rules are all code-based. If you ever get any of these values from glCheckFramebufferStatus​, it is because your program has done something wrong in setting up the FBO. Each one has a specific remedy for it.

There's one more rule that can trip you up:

• The implementation likes your combination of attached image formats. (GL_FRAMEBUFFER_UNSUPPORTED when false).

OpenGL allows implementations to state that they do not support some combination of image formats for the attached images; they do this by returning GL_FRAMEBUFFER_UNSUPPORTED when you attempt to use an unsupported format combination.

However, the OpenGL specification also requires that implementations support certain format combinations; if you use these, implementations are forbidden to return GL_FRAMEBUFFER_UNSUPPORTED. Implementations must allow any combination of color formats, so long as all of those color formats come from the required set of color formats.

These color formats can be combined with a depth attachment with any of the required depth formats. Stencil attachments can also be used, again with the required stencil formats, as well as the combined depth/stencil formats. However, implementations are only required to support both a depth and stencil attachment simultaneously if both attachments refer to the same image.

This means that if you want stencil with no depth, you can use one of the required stencil formats. If you want depth with no stencil, you can use one of the required depth formats. But if you want depth and stencil, you must use a depth/stencil format and the same image in that texture must be attached to both the depth and stencil attachments.

Staying within these limits means you won't see GL_FRAMEBUFFER_UNSUPPORTED. Going outside of these limits makes it entirely possible to get this incompleteness.

Wow!  That’s a lot of rules to follow and a lot of head scratching can occur when you forget just one tiny thing.  This makes using FBOs, if not difficult, then at least tedious.

QOpenGLFramebufferObject

Fortunately, Qt comes to our rescue again and provides a platform-independent/driver-independent wrapper for FBOs that make them incredibly easy to use. The most common case, render-to-texture, can be accomplished in just 2 lines of code.

That’s it.  The first line creates the framebuffer object, then creates a texture object with an associated texture-image sized 640 x 480, then creates a combine Depth/Stencil render buffer sized 640-480, attaches them the GL_COLOR_ATTACHMENT0 and GL_DEPTH_STENCIL_ATTACHMENT respectively, and validates FBO completeness with glCheckFrameBufferStatus().    The second line makes this FBO “active” and you’re ready to render-to-texture.  Once you’ve finished rendering-to-texture unbind the FBO with myFBO.release() and get the texture object Qt created for you via myFBO.texture() and use it anyway you like in further rendering calls.

Let’s recreate the pbuffer example from the first part of this article but this time using Modern OpenGL with FBOs and Qt for our platform layer.

Listing 6.2 shows the complete program.

Wrapping Up

We covered quite a lot of material this time around but now we’re ready for a whole host of advanced rendering techniques.  Next time we’ll use FBO extensively when I cover how to do “picking” with Modern OpenGL. Until then, it’s been a pleasure writing for you and I hope you come back soon.

Reference

1. OpenGL website page, last accessed April 13, 2016, https://www.opengl.org/wiki/Framebuffer_Object
2. OpenGL website page, last accessed April 13, 2016, https://www.opengl.org/registry/specs/ARB/wgl_render_texture.txt