::graphics and programming

Improving the look of a large image in a small UIImageView with an one liner

It is not always that an image view and its image have the same size. Often times the size of the images we obtain from a web service don’t match the size of the image views where they are displayed. Or you might present an image picker controller so the user can take a photo and then display this photo in a thumbnail before proceeding to a next step.

Consider the following photo:

Mercedes AMG GTS

This is what you get when you display it in a small image view:

The result is grainy, noisy. You could obviously resize the original image and use the smaller version in this small image view instead. That would give you a smoother result. That sounds like a bit of work though, especially considering that you can obtain an equivalent result with one simple line of code:

imageView.layer.minificationFilter = kCAFilterTrilinear;

And we get:

Now that doesn’t hurt the eyes like the previous image. It looks as smooth as we would like.

Now let’s discuss how all this works.

To draw an image on the screen, the pixels of the image must be mapped to the pixels of the screen. This process is known as filtering. The image can be filtered in a variety of ways and the most popular algorithms have a quality vs speed tradeoff as usual. This filtering, however, is performed by the hardware and the hardware only implements the most basic algorithms that are usually good enough and very fast.

The simplest one is the nearest filter (kCAFilterNearest). For each pixel on the screen it just picks the color of one of the pixels of the image that is located under that screen pixel, usually the image pixel that is closest to the middle of the screen pixel. The following image shows the same photo scaled down to 32×24 pixels using a nearest filter:

The result is a mess, especially because the original image is much bigger than the resized image.

The default filter on iOS is the bilinear filter (kCAFilterLinear). It selects 4 pixels in the image that are closest to the center of the screen pixel and computes an weighted average of the color of these pixels as the output. The article Bilinear Texture Filtering in the Direct3D documentation gives a nice explanation on bilinear filtering. If we scale down the same photo to 32×24 using a bilinear filter we get this:

 The result is very similar to the nearest filter because there is a big difference between the original image size and the filtered image size. The bilinear filter only does a really nice job if the filtered image is not much smaller than half the size of the original image. The filtered image becomes grainer as it shrinks.

The filter this post highlights is the trilinear filter (kCAFilterTrilinear). Before the trilinear filter can be used, a mipmap must be generated for the original image. The mipmap is a sequence of smaller versions of the original image, where each subsequent image has half the width and height of the original. The last item in the sequence is a 1×1 image. The mipmap is generated for you by the hardware.

The trilinear filter dynamically selects one image in the mipmap array that is more appropriate for the image view size and then applies a bilinear filter on it. That also makes images look perfect at all times in image views where you run scaling animations. Here is the same photo scaled down to 32×24 using a trilinear filter:

It looks really smooth even for such extreme down scaling.

Storing mipmaps for your images requires more memory, of course. However, that is usually not a problem since the mipmap adds about 1/3 of the size of the original image to the memory. That would only become significant if you’re working with really large images.

Note that if you’re displaying a large image on a small UIButton, you should set the minificationFilter property to kCAFilterTrilinear on its imageView.layer, not on the button itself.

Share

Presenting Multiple Modal View Controllers at Once

It’s quite unusual, but eventually you might need to present multiple modal View Controllers at once, where it looks like only the top most View Controller (VC) is being presented. The others will appear as the user dismisses the topmost VC. For instance, you might want to present an UIImagePickerController to take/choose a picture and show another VC to edit the image after the UIImagePickerController is dismissed.

The trivial way to attempt to achieve that is to just create the VCs you want to present modally and present one after the other. Unfortunately, this is not gonna work. What you get is just the first VC presented, all others just go nowhere. UIKit just won’t cooperate with you here. You might try some variations of this, but I am almost certain none of them are going to work as you would expect. However, there’s still one trick that will make that happen…

If you present the first VC without animation and then present the second with animation, the first will appear suddenly and then the second will be presented with animation. But we don’t want for the first VC to appear. We want for it to be transparent until the animation of the presentation of the second VC completes. So, let’s set the alpha property of its view to zero before presenting it, and then set it back to 1 in the completion block of the presentViewController:animated:completion:. Now what we get is a black background during the presentation of the second VC. This happens because of an optimization in UIKit where it hides the VCs that are under the topmost VC, and it does not take the alpha into account. Since we are presenting a VC modally without animation, we see either the window background color or whatever root view controller that might be in it, since the VC’s view alpha is zero. So what we can do to keep our current VC on the background while the second VC is animating into the screen? Yes, you might have thought of that: put an image with the contents of the current VC on the background.

So the outline of the method should be as follows:

  1. Create VCs A and B, where B will be on the top.
  2. Draw the current VC on an image through -[CALayer renderInContext:].
  3. Create an imageView with this image and insert it in the main window at index 0 so it will be behind everything.
  4. Set A’s view alpha to zero.
  5. Present B using presentViewController:animated:completion: and set A’s view alpha to one in the completion block and also remove the imageView from its superview (the image will be deallocated in this step).

And here is some sample code:

UIViewController *aViewController = /*_____*/; // Create your VC somehow
UIViewController *bViewController = /*______*/;

// Create the image
UIGraphicsBeginImageContextWithOptions(self.view.bounds.size, NO, 0);
CGContextRef context = UIGraphicsGetCurrentContext();
[self.view.layer renderInContext:context];
UIImage *image = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
UIImageView *imageView = [[UIImageView alloc] initWithFrame:self.view.frame];
imageView.image = image
[self.window insertSubview:imageView atIndex:0];

// Present the VCs
aViewController.view.alpha = 0; // Set alpha to zero so that we can see the imageView
[self presentViewController:aViewController animated:NO completion:nil];
[self presentViewController:bViewController animated:YES completion:^{
    aViewController.view.alpha = 1;
    [imageView removeFromSuperview];
}];

Then, after dismissing bViewController, you should see aViewController on the screen.

Share

Faster Gaussian Blur in GLSL

The popular technique to make a gaussian blur in GLSL is a two-pass blur, in the horizontal and then in the vertical direction (or vice-versa), also known as a convolution. This is a classic post on this matter http://www.gamerendering.com/2008/10/11/gaussian-blur-filter-shader/, and many others you will find around the web do exactly the same. But the performance of this shader can be improved quite a lot.

The bottleneck is in the dependent texture reads, which are any texture reads in a fragment shader where the coordinates depend on some calculation, or any texture reads in a vertex shader. A non-dependent texture read is one where the coordinates are either a constant or a varying. According to the PowerVR Performance Recommendations, a dependent texture read adds a lot of additional steps in the processing of your shader because the texture coordinates cannot be known ahead of time. When you use a constant or a varying, the hardware is able to pre-fetch the texture data before it gets to run your fragment shader, hence it becomes way more efficient.

In this gaussian blur shader we can get completely rid of the dependent reads. The key is to compute all the texture coordinates in the vertex shader and store them in varyings. They will be automatically interpolated as usual and you will have the benefits of the texture pre-fetch that occurs before the GPU runs the fragment shader. In GLSL ES we can use up to 32 floats in our varyings, which allow us to pre-compute up to 16 vec2 texture coordinates in the vertex shader.

We need two vertex shaders, one for the first pass which computes the horizontal texture coordinates and another for the second pass which computes the vertical texture coordinates. And we need a single fragment shader that we can use in both passes. This is what the first vertex shader looks like:

/* HBlurVertexShader.glsl */
attribute vec4 a_position;
attribute vec2 a_texCoord;

varying vec2 v_texCoord;
varying vec2 v_blurTexCoords[14];

void main()
{
    gl_Position = a_position;
    v_texCoord = a_texCoord;
    v_blurTexCoords[ 0] = v_texCoord + vec2(-0.028, 0.0);
    v_blurTexCoords[ 1] = v_texCoord + vec2(-0.024, 0.0);
    v_blurTexCoords[ 2] = v_texCoord + vec2(-0.020, 0.0);
    v_blurTexCoords[ 3] = v_texCoord + vec2(-0.016, 0.0);
    v_blurTexCoords[ 4] = v_texCoord + vec2(-0.012, 0.0);
    v_blurTexCoords[ 5] = v_texCoord + vec2(-0.008, 0.0);
    v_blurTexCoords[ 6] = v_texCoord + vec2(-0.004, 0.0);
    v_blurTexCoords[ 7] = v_texCoord + vec2( 0.004, 0.0);
    v_blurTexCoords[ 8] = v_texCoord + vec2( 0.008, 0.0);
    v_blurTexCoords[ 9] = v_texCoord + vec2( 0.012, 0.0);
    v_blurTexCoords[10] = v_texCoord + vec2( 0.016, 0.0);
    v_blurTexCoords[11] = v_texCoord + vec2( 0.020, 0.0);
    v_blurTexCoords[12] = v_texCoord + vec2( 0.024, 0.0);
    v_blurTexCoords[13] = v_texCoord + vec2( 0.028, 0.0);
}

As you can see, the v_blurTexCoords varying is an array of vec2 where we store a bunch of vectors around the texture coordinate of the vertex a_texCoord. Each vec2 in this array will be interpolated along the triangles being rendered, as expected. We do the same for the other vertex shader which pre-computes the vertical blur texture coordinates:

/* VBlurVertexShader.glsl */
attribute vec4 a_position;
attribute vec2 a_texCoord;

varying vec2 v_texCoord;
varying vec2 v_blurTexCoords[14];

void main()
{
    gl_Position = a_position;
    v_texCoord = a_texCoord;
    v_blurTexCoords[ 0] = v_texCoord + vec2(0.0, -0.028);
    v_blurTexCoords[ 1] = v_texCoord + vec2(0.0, -0.024);
    v_blurTexCoords[ 2] = v_texCoord + vec2(0.0, -0.020);
    v_blurTexCoords[ 3] = v_texCoord + vec2(0.0, -0.016);
    v_blurTexCoords[ 4] = v_texCoord + vec2(0.0, -0.012);
    v_blurTexCoords[ 5] = v_texCoord + vec2(0.0, -0.008);
    v_blurTexCoords[ 6] = v_texCoord + vec2(0.0, -0.004);
    v_blurTexCoords[ 7] = v_texCoord + vec2(0.0,  0.004);
    v_blurTexCoords[ 8] = v_texCoord + vec2(0.0,  0.008);
    v_blurTexCoords[ 9] = v_texCoord + vec2(0.0,  0.012);
    v_blurTexCoords[10] = v_texCoord + vec2(0.0,  0.016);
    v_blurTexCoords[11] = v_texCoord + vec2(0.0,  0.020);
    v_blurTexCoords[12] = v_texCoord + vec2(0.0,  0.024);
    v_blurTexCoords[13] = v_texCoord + vec2(0.0,  0.028);
}

The fragment shader just computes the gaussian-weighted average of the color of the texels on these pre-computed texture coordinates. Since it uses the generic v_blurTexCoords array, we can use this same fragment shader in both passes.

/* BlurFragmentShader.glsl */
precision mediump float;

uniform sampler2D s_texture;

varying vec2 v_texCoord;
varying vec2 v_blurTexCoords[14];

void main()
{
    gl_FragColor = vec4(0.0);
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 0])*0.0044299121055113265;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 1])*0.00895781211794;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 2])*0.0215963866053;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 3])*0.0443683338718;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 4])*0.0776744219933;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 5])*0.115876621105;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 6])*0.147308056121;
    gl_FragColor += texture2D(s_texture, v_texCoord         )*0.159576912161;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 7])*0.147308056121;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 8])*0.115876621105;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[ 9])*0.0776744219933;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[10])*0.0443683338718;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[11])*0.0215963866053;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[12])*0.00895781211794;
    gl_FragColor += texture2D(s_texture, v_blurTexCoords[13])*0.0044299121055113265;
}

As you can see, all the texture reads (calls to the built-in texture2D function) receive a pre-computed texture coordinate, that is why the hardware is able to pre-fetch the texture data for these texture reads, which is a huge improvement. For more details about what happens in the hardware, check out the section 5 of the PowerVR SGX Architecture Guide for Developers.

Then, to render the blurred image we draw into an off-screen buffer (Render toTexture) the first pass using the HBlurVertexShader and the BlurFragmentShader and the second pass using the VBlurVertexShader and the BlurFragmentShader. Of course the second pass might be rendered directly into the main frame buffer if no more post processing is going to take place.

Obviously, this technique can also be applied in any other similar scenarios. You can find an implementation of blur and sharpen filters in XBImageFilters.

Share

Eliminating branches in shaders

You might have heard GPUs hate branches (if-elses, and consequently dynamic for- and while-loops). See more details here http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter34.html. Instead of trying to figure out how to best setup your branches, how about getting completely rid of them? That is possible and easy to do in most cases.

I am gonna show you how to eliminate branches in shaders with one example. That is usually done using GLSL built-in functions like mix() (lerp() in HLSL), clamp(), sign() and abs(). In this example I will write a fragment shader that “overlays” one texture on top of another. In this case the overlay is the Photoshop Overlay Blending Mode. I am using the formulas from this classic post http://mouaif.wordpress.com/2009/01/05/photoshop-math-with-glsl-shaders/. The naïve implementation of the Overlay blending requires one branch for each of the RGB channels. Basically, what it does is the following: for each channel, if the value in the base image is smaller than 0.5, return 2 * base * blend; otherwise, return 1 - 2 * (1 - base) * (1 - blend). Where blend is the image on top of the base image. Conceptually, if the value is lower than 0.5, multiply, else screen it. Then, the fragment shader would end up looking like this:

// Note: This is GLSL ES
precision highp float;

uniform sampler2D s_texture; // Base texture
uniform sampler2D s_overlay; // Overlay texture

varying vec2 v_texCoord;

float overlayf(float base, float blend)
{
    if (base < 0.5) {
        return 2.0 * base * blend;
    }
    else {
        return 1.0 - 2.0 * (1.0 - base) * (1.0 - blend);
    }
}

vec3 overlay(vec3 base, vec3 blend)
{
    return vec3(overlayf(base.r, blend.r), overlayf(base.g, blend.g), overlayf(base.b, blend.b));
}

void main()
{
    vec4 base = texture2D(s_texture, v_texCoord);
    vec4 blend = texture2D(s_overlay, v_texCoord);
    vec3 overlay = overlay(base.rgb, blend.rgb); // Overlay'd color
    // Linearly interpolate between the base color and overlay'd color
    // because the blend texture might have transparency. The built-in
    // mix does the job since mix(x, y, a) = x*(1.0 - a) + y*a
    vec3 finalColor = mix(base.rgb, overlay.rgb, blend.a);
    gl_FragColor = vec4(finalColor, base.a);
}

That’s quite a lot of code.. It’s possible to make it much simpler with the smart use of the built-in functions. The general technique is to convert the comparison (base < 0.5) into a float which is zero if base is below 0.5, and is one otherwise. Then we use this float as the interpolation parameter (the third) in the mix() built-in. It will allow us to select between the first and the second parameters since it is either zero (then mix() returns the first parameter) or one (mix() returns the second parameter), for example:

float a = 0.0;
float x = 1.0;
float y = 2.0;
float z = mix(x, y, a); // z will be equals to x
float b = 1.0;
float k = mix(x, y, b); // k will be equals to y

Then, our final overlay shader code will be:

precision highp float;

uniform sampler2D s_texture;
uniform sampler2D s_overlay;

varying vec2 v_texCoord;
varying vec2 v_rawTexCoord;

void main()
{
    vec4 base = texture2D(s_texture, v_texCoord);
    vec4 blend = texture2D(s_overlay, v_texCoord);
    // This is our 'selection parameter'. For each of the rgb components, we subtract 0.5
    // and take the sign() of the result. It will return -1.0 if smaller than 0.0, 0.0 if 0.0,
    // and 1.0 if greater than 0.0. In this case, we want for the final result to be 0.0
    // when the subtraction is smaller than zero and 1.0 otherwise, then we clamp() the value
    // in the [0.0, 1.0] interval. Hence, when smaller than zero, it will return -1.0 and this
    // value will be clamped to 0.0.
    vec3 br = clamp(sign(base.rgb - vec3(0.5)), vec3(0.0), vec3(1.0));
    vec3 multiply = 2.0 * base.rgb * blend.rgb;
    vec3 screen = vec3(1.0) - 2.0 * (vec3(1.0) - base.rgb)*(vec3(1.0) - blend.rgb);
    // If br is 0.0, overlay will be multiply (which translates to if (base < 0.5) { return multiply; }).
    // if bt is 1.0, overlay will be screen (which translates to if (base >= 0.5) { return screen; }).
    vec3 overlay = mix(multiply, screen, br);
    vec3 finalColor = mix(base.rgb, overlay, blend.a);
    gl_FragColor = vec4(finalColor, base.a);
}

Now the code is much shorter and cleaner and performs exactly the same function. The difference is that we compute both the values of the if and the else and then select one of them. When branching, we avoid wasting the computation of the unused value, which in this example has a negligible cost. On the other hand, we have no branches, which keeps the GPU happy. Depending on your hardware and shader complexity, you might notice a performance improvement.

Share

UIView – Frame and Transform

You should be careful when messing around with the transform property of a view. Usually, when doing it you should not touch the frame property of the view (unless you know what you’re doing), because the frame is the axis-aligned bounding rectangle (AABR) of a view (also known as axis-aligned bounding box). Then, it changes whenever you change the transform of your view, and this change might look really weird for you if you don’t know how this works, which is what I am gonna show in this post.

The AABR is the smallest rectangle aligned with the coordinate axes (the x (horizontal) and y (vertical) axes) that contains a given shape, in this case, an UIView instance. So, if you rotate the view a little, the previous AABR won’t anymore contain the view completely, then it must change. I have setup a simple example project on Github to display this relationship between the frame and the transform of an UIView, which you can find here https://github.com/xissburg/FrameTransform. Here we can see two screenshots of it:

Basically, it contains one view in the front (dark gray) and another in the back (light gray). The view in the front is transformed by gestures (pan, pinch and rotate) and after the transform is applied the frame of the front view is assigned to the frame of the back view. In the previous image, we have on the left the front view in its initial configuration, no transforms applied. On the right, a small rotation was applied and we can see what its frame looks like now (it is the light gray rectangle). It has increased to accommodate the new shape of the front view. When it is scaled the same thing happens, the frame is also scaled to accommodate the view again.

The conclusion is, be careful when changing both the transform and the frame of a view. I am not saying you should not do that, because there are a few situations where it is handy to change both to obtain the result you want. That is why in most cases you should use the bounds of the view instead, because the bounds don’t change under transforms.

Share

iOS UI Design – A Few Dos and Don’ts

In this post I want to address a few mistakes designers and programmers often make when designing user interfaces for iOS apps. These are related to incorrect image sizes for the retina display that leads to small misalignments, which kills your app on the eyes of a person who appreciates details.

First of all, this is something I always say: Designers must be, at least, perfectionists. I am not talking about polishing a single pixel for days, not an obsessive perfectionist, which usually is another problem. I am talking about having a sharp, critical eye on a design, to be able to spot minor mistakes and misalignments, which is essential in UI design where things must be aligned properly. I have worked with many designers, and they did a good-looking work, but while coding the corresponding app I would often find misaligned UI components, others that should be centered horizontally but were not, or had an odd width. If something have to be centered horizontally, it must be pixel-perfectly centered horizontally, period. On the other side, the design might have a perfect layout, but the programmer is the one who gives life to the app. Programmers are usually not designers neither have an eye for such level of detail, then they might introduce errors in the layout that was initially perfect.

Are the buttons vertically aligned one with the other?

To notice such small details at a first glance is a skill that you can only develop by practicing. Look at the design, try to notice something that is not in harmony and then measure it. I like to measure things in Photoshop using the Info window and the Rectangular Marquee tool. The Ruler tool might also be useful.

The Info window shows the width and height of the rectangular selection.

 Image sizes for the retina display

Because our apps run on retina and non-retina displays we have to provide images for both displays, one being half the size of the other. Notice the word half. Pixels are not divisible. There’s no such thing as ‘one third of a pixel’. That is why the sides of any image for the retina display must be even (one exception would be 1 pixel wide gradients, for example). If you ask for Photoshop to resize a 101x57px image to 50%, you get a 51x29px image, which is not half of the other, and this kind of uneven resizing also leads to small resampling artifacts. Then, always keep the size of your UI components even in both dimensions for the retina display.

There’s one more requirement for UI components that are centered on the screen. Suppose you have a 150x80px image that you want to center horizontally on the screen. On the non-retina display this image will have 75x40px. What is the problem? The width in the non-retina display is odd. Despite the retina display having twice the resolution of the non-retina, it actually has the same size but twice the scale. That is why we specify positions and sizes in the non-retina coordinate system (width 320, height 480). Then, if you’re going to center a view that is 75 pixels wide, well, you won’t be able to precisely center it actually. You either leave on more column of pixels on the right or the left. That is why the width of an image you want to center horizontally must be a multiple of 4, so that after you divide by 2 it will still be even. In this example, the designer could make it either 152x80px or 148x80px.

The location of pixels plays a role in their look in the resized image

When an image is resized it is resampled/filtered. In this process the old image (the large one if downsizing) is projected on the new image and the color of each pixel of the new image is determined by averaging the color of the pixels in the old image around the center of the new pixel, using some weight function to average the values. Hence, the location of a stroke or straight line will affect its look in the resized image, which might be blurry if not well placed.

For example, lets resize two variations of a simple button wannabe, originally 80x60px for the retina display:

Retina display button at 5x zoom.

Note that in this image the black borders of the button are 2 pixels away from the image borders. After resizing this image to 50% we get:

Same button from the previous image, resized to 50%, zoomed 10x.

As you can see, the borders are consistent and sharp because they are in a good location in the original retina display image. The borders are at even locations, then when resampling they match a row of pixels perfectly in the low resolution image, and there’s no blurring.

Now, let’s try a different alignment for the borders of this button. The only difference in the following button is that the borders of the button are now 1 pixel away from the border of the image.

The same button with borders a bit misaligned.

Now the borders won’t match full rows in the small image.

Blurred borders because of misalignment.

And we have blurred borders, which look pretty bad especially when zoomed in.

Then, always try to keep your sharp straight lines in an even coordinate. Also, usually when you want to make a 1 pixel wide line, you should actually make it 2 pixels wide because of resizing.

The retina display is not big

Because of the resolution of the retina display, designers sometimes forget that the screen is as big as the non-retina display on the device, and seem to think that it is as big as a 640x960px rectangle in a regular screen, which is almost as big as an iPad screen. This leads them to create undersized UI elements, which they (or the programmer) only notice when the programmer starts to implement the app. It’s too late then. To avoid this serious problem, always keep an eye on your design at 50% of the original resolution, which is exactly the size of the non-retina display. In Photoshop, you can create another window for the same image in the menu Window > Arrange > New Window for [your file name], zoom out to 50% and keep an eye on it to see what the actual size looks like.

Share