Right now our browser can only draw colored rectangles and text—pretty boring! Real browsers support all kinds of visual effects that change how pixels and colors blend together. Let’s implement these effects using the Skia graphics library, and also see a bit of how Skia is implemented under the hood. That’ll also allow us to use surfaces for browser compositing to accelerate scrolling.

Installing Skia and SDL

Before we get any further, we’ll need to upgrade our graphics system. While Tkinter is great for basic shapes and handling input, it lacks built-in support for many visual effects.That’s because Tk, the graphics library that Tkinter uses, dates from the early 90s, before high-performance graphics cards and GPUs became widespread. Implementing fast visual effects routines is fun, but it’s outside the scope of this book, so we need a new graphics library. Let’s use Skia, the library that Chromium uses. Unlike Tkinter, Skia doesn’t handle inputs or create graphical windows, so we’ll pair it with the SDL GUI library.

Start by installing Skia and SDL:

pip3 install skia-python pysdl2 pysdl2-dll

Once installed, remove the tkinter imports from browser and replace them with these:

import ctypes
import sdl2
import skia

If any of these imports fail, you probably need to check that Skia and SDL were installed correctly. Note that the ctypes module comes standard in Python; it is used to convert between Python and C types.

SDL creates the window

The main loop of the browser first needs some boilerplate to get SDL started:

if __name__ == "__main__":
    import sys
    sdl2.SDL_Init(sdl2.SDL_INIT_EVENTS)
    browser = Browser()
    browser.load(sys.argv[1])
    # ...

Next, we need to create an SDL window, instead of a Tkinter window, inside the Browser, and set up Skia to draw to it. Here’s the SDL incantation to create a window:

class Browser:
    def __init__(self):
        self.sdl_window = sdl2.SDL_CreateWindow(b"Browser",
            sdl2.SDL_WINDOWPOS_CENTERED, sdl2.SDL_WINDOWPOS_CENTERED,
            WIDTH, HEIGHT, sdl2.SDL_WINDOW_SHOWN)

To set up Skia to draw to this window, we also need create a surface for it:In Skia and SDL, a surface is a representation of a graphics buffer into which you can draw pixels (bits representing colors). A surface may or may not be bound to the physical pixels on the screen via a window, and there can be many surfaces. A canvas is an API interface that allows you to draw into a surface with higher-level commands such as for rectangles or text. Our browser uses separate Skia and SDL surfaces for simplicity, but in a highly optimized browser, minimizing the number of surfaces is important for good performance.

class Browser:
    def __init__(self):
        self.root_surface = skia.Surface.MakeRaster(
            skia.ImageInfo.Make(
                WIDTH, HEIGHT,
                ct=skia.kRGBA_8888_ColorType,
                at=skia.kUnpremul_AlphaType))

Typically, we’ll draw to the Skia surface, and then once we’re done with it we’ll copy it to the SDL surface to display on the screen. This will be a little hairy, because we are moving data between two low-level libraries, but really it’s just copying pixels from one place to another.

First, get the sequence of bytes representing the Skia surface:

class Browser:
    def draw(self):
        # ...

        # This makes an image interface to the Skia surface, but
        # doesn't actually copy anything yet.
        skia_image = self.root_surface.makeImageSnapshot()
        skia_bytes = skia_image.tobytes()

Next, we need to copy the data to an SDL surface. This requires telling SDL what order the pixels are stored in (which we specified to be RGBA_8888 when constructing the surface) and on your computer’s endianness:

class Browser:
    def __init__(self):
        if sdl2.SDL_BYTEORDER == sdl2.SDL_BIG_ENDIAN:
            self.RED_MASK = 0xff000000
            self.GREEN_MASK = 0x00ff0000
            self.BLUE_MASK = 0x0000ff00
            self.ALPHA_MASK = 0x000000ff
        else:
            self.RED_MASK = 0x000000ff
            self.GREEN_MASK = 0x0000ff00
            self.BLUE_MASK = 0x00ff0000
            self.ALPHA_MASK = 0xff000000

The CreateRGBSurfaceFrom method then wraps the data in an SDL surface (this SDL surface does not copy the bytes): Note that since Skia and SDL are C++ libraries, they are not always consistent with Python’s garbage collection system. So the link between the output of tobytes and sdl_window is not guaranteed to be kept consistent when skia_bytes is garbage collected. Instead, the SDL surface will be pointing at a bogus piece of memory, which will lead to memory corruption or a crash. The code here is correct because all of these are local variables that are garbage-collected together, but if not you need to be careful to keep all of them alive at the same time.

class Browser:
    def draw(self):
        # ...
        depth = 32 # Bits per pixel
        pitch = 4 * WIDTH # Bytes per row
        sdl_surface = sdl2.SDL_CreateRGBSurfaceFrom(
            skia_bytes, WIDTH, HEIGHT, depth, pitch,
            self.RED_MASK, self.GREEN_MASK,
            self.BLUE_MASK, self.ALPHA_MASK)

Finally, we draw all this pixel data on the window itself by blitting (copying) it from sdl_surface to sdl_window’s surface:

class Browser:
    def draw(self):
        # ...
        rect = sdl2.SDL_Rect(0, 0, WIDTH, HEIGHT)
        window_surface = sdl2.SDL_GetWindowSurface(self.sdl_window)
        # SDL_BlitSurface is what actually does the copy.
        sdl2.SDL_BlitSurface(sdl_surface, rect, window_surface, rect)
        sdl2.SDL_UpdateWindowSurface(self.sdl_window)

Next, SDL doesn’t have a mainloop or bind method; we have to implement it ourselves:

if __name__ == "__main__":
    # ...
    event = sdl2.SDL_Event()
    while True:
        while sdl2.SDL_PollEvent(ctypes.byref(event)) != 0:
            if event.type == sdl2.SDL_QUIT:
                browser.handle_quit()
                sdl2.SDL_Quit()
                sys.exit()
            # ...

The details of ctypes and PollEvent aren’t too important here, but note that SDL_QUIT is an event, sent when the user closes the last open window. The handle_quit method it calls just cleans up the window object:

class Browser:
    def handle_quit(self):
        sdl2.SDL_DestroyWindow(self.sdl_window)

We’ll also need to handle all of the other events in this loop—clicks, typing, and so on. The SDL syntax looks like this:

if __name__ == "__main__":
    while True:
        while sdl2.SDL_PollEvent(ctypes.byref(event)) != 0:
            # ...
            elif event.type == sdl2.SDL_MOUSEBUTTONUP:
                browser.handle_click(event.button)
            elif event.type == sdl2.SDL_KEYDOWN:
                if event.key.keysym.sym == sdl2.SDLK_RETURN:
                    browser.handle_enter()
                elif event.key.keysym.sym == sdl2.SDLK_DOWN:
                    browser.handle_down()
            elif event.type == sdl2.SDL_TEXTINPUT:
                browser.handle_key(event.text.text.decode('utf8'))

I’ve changed the signatures of the various event handler methods; you’ll need to make analogous changes in Browser where they are defined. This loop replaces all of the bind calls in the Browser constructor, which you can now remove.

Skia provides the canvas

Now our browser is creating an SDL window and can draw to it via Skia. But most of the browser codebase is still using Tkinter drawing commands, which we now need to replace. Skia is a bit more verbose than Tkinter, so let’s abstract over some details with helper functions.Consult the Skia and skia-python documentation for more on the Skia API. First, a helper function to convert colors to Skia colors:

def parse_color(color):
    if color == "white":
        return skia.ColorWHITE
    elif color == "lightblue":
        return skia.ColorSetARGB(0xFF, 0xAD, 0xD8, 0xE6)
    # ...
    else:
        return skia.ColorBLACK

You can add more “elif” blocks to support any other color names you use; modern browsers support quite a lot.

To draw a line, you use Skia’s Path object:

def draw_line(canvas, x1, y1, x2, y2):
    path = skia.Path().moveTo(x1, y1).lineTo(x2, y2)
    paint = skia.Paint(Color=skia.ColorBLACK)
    paint.setStyle(skia.Paint.kStroke_Style)
    paint.setStrokeWidth(1)
    canvas.drawPath(path, paint)

To draw text, you use drawString:

def draw_text(canvas, x, y, text, font, color=None):
    sk_color = parse_color(color)
    paint = skia.Paint(AntiAlias=True, Color=sk_color)
    canvas.drawString(
        text, float(x), y - font.getMetrics().fAscent,
        font, paint)

Finally, for drawing rectangles you use drawRect:

def draw_rect(canvas, l, t, r, b, fill=None, width=1):
    paint = skia.Paint()
    if fill:
        paint.setStrokeWidth(width)
        paint.setColor(parse_color(fill))
    else:
        paint.setStyle(skia.Paint.kStroke_Style)
        paint.setStrokeWidth(1)
        paint.setColor(skia.ColorBLACK)
    rect = skia.Rect.MakeLTRB(l, t, r, b)
    canvas.drawRect(rect, paint)

If you look at the details of these helper methods, you’ll see that they all use a Skia Paint object to describe a shape’s borders and colors. We’ll be seeing a lot more features of Paint in this chapter.

With these helper methods we can now upgrade our browser’s drawing commands to use Skia:

class DrawText:
    def execute(self, scroll, canvas):
        draw_text(canvas, self.left, self.top - scroll,
            self.text, self.font, self.color)

class DrawRect:
    def execute(self, scroll, canvas):
        draw_rect(canvas,
            self.left, self.top - scroll,
            self.right, self.bottom - scroll,
            fill=self.color, width=0)

Let’s also add a rect field to each drawing command, replacing its top, left, bottom, and right fields with a Skia Rect object:

class DrawText:
    def __init__(self, x1, y1, text, font, color):
        # ...
        self.rect = \
            skia.Rect.MakeLTRB(x1, y1, self.right, self.bottom)

class DrawRect:
    def __init__(self, x1, y1, x2, y2, color):
        # ...
        self.rect = skia.Rect.MakeLTRB(x1, y1, x2, y2)

Finally, the Browser class also uses Tkinter commands in its draw method to draw the browser UI. We’ll need to change them all to use Skia. It’s a long method, so we’ll need to go step by step.

First, clear the canvas and and draw the current Tab into it:

class Browser:
    def draw(self):
        canvas = self.root_surface.getCanvas()
        canvas.clear(skia.ColorWHITE)

        self.tabs[self.active_tab].draw(canvas)

Then draw the browser UI elements. First, the tabs:

class Browser:
    def draw(self):
        # ...
        tabfont = skia.Font(skia.Typeface('Arial'), 20)
        for i, tab in enumerate(self.tabs):
            name = "Tab {}".format(i)
            x1, x2 = 40 + 80 * i, 120 + 80 * i
            draw_line(canvas, x1, 0, x1, 40)
            draw_line(canvas, x2, 0, x2, 40)
            draw_text(canvas, x1 + 10, 10, name, tabfont)
            if i == self.active_tab:
                draw_line(canvas, 0, 40, x1, 40)
                draw_line(canvas, x2, 40, WIDTH, 40)

Next, the plus button for adding a new tab:I also changed the y position of the plus sign. Skia draws fonts a bit differently from Tkinter, and the new y position keeps the plus centered in the box. Feel free to adjust the positions of the UI elements to make everything look good on your system.

class Browser:
    def draw(self):
        # ...
        buttonfont = skia.Font(skia.Typeface('Arial'), 30)
        draw_rect(canvas, 10, 10, 30, 30)
        draw_text(canvas, 11, 4, "+", buttonfont)

Then the address bar, including text and cursor:

class Browser:
    def draw(self):
        # ...
        draw_rect(canvas, 40, 50, WIDTH - 10, 90)
        if self.focus == "address bar":
            draw_text(canvas, 55, 55, self.address_bar, buttonfont)
            w = buttonfont.measureText(self.address_bar)
            draw_line(canvas, 55 + w, 55, 55 + w, 85)
        else:
            url = self.tabs[self.active_tab].url
            draw_text(canvas, 55, 55, url, buttonfont)

And finally the “back” button:

class Browser:
    def draw(self):
        # ...
        draw_rect(canvas, 10, 50, 35, 90)
        path = \
            skia.Path().moveTo(15, 70).lineTo(30, 55).lineTo(30, 85)
        paint = skia.Paint(
            Color=skia.ColorBLACK, Style=skia.Paint.kFill_Style)
        canvas.drawPath(path, paint)

Tab also has a draw method, which draws a cursor; it needs to use draw_line for that. Also wrap it in a display list command called DrawLine.

class DrawLine:
    def __init__(self, x1, y1, x2, y2):
        self.rect = skia.Rect.MakeLTRB(x1, y1, x2, y2)
        self.x1 = x1
        self.y1 = y1
        self.x2 = x2
        self.y2 = y2

    def execute(self, canvas):
        draw_line(canvas, self.x1, self.y1, self.x2, self.y2)

class Tab:
    def render(self):
        # ...
        if self.focus:
            obj = [obj for obj in tree_to_list(self.document, [])
               if obj.node == self.focus and \
                    isinstance(obj, InputLayout)][0]
            text = self.focus.attributes.get("value", "")
            x = obj.x + obj.font.measureText(text)
            y = obj.y
            self.display_list.append(
                DrawLine(x, y, x, y + obj.height))

That’s most of it. The last few changes we need to upgrade from Tkinter to SDL and Skia relate to fonts and text.

Skia is also the font library

Since we’re replacing Tkinter with Skia, we are also replacing tkinter.font. In Skia, a font object has two pieces: a Typeface, which is a type family with a certain weight, style, and width; and a Font, which is a Typeface at a particular size. It’s the Typeface that contains data and caches, so that’s what we need to cache:

def get_font(size, weight, style):
    key = (weight, style)
    if key not in FONTS:
        if weight == "bold":
            skia_weight = skia.FontStyle.kBold_Weight
        else:
            skia_weight = skia.FontStyle.kNormal_Weight
        if style == "italic":
            skia_style = skia.FontStyle.kItalic_Slant
        else:
            skia_style = skia.FontStyle.kUpright_Slant
        skia_width = skia.FontStyle.kNormal_Width
        style_info = \
            skia.FontStyle(skia_weight, skia_width, skia_style)
        font = skia.Typeface('Arial', style_info)
        FONTS[key] = font
    return skia.Font(FONTS[key], size)

Our browser also needs font metrics and measurements. In Skia, these are provided by the measureText and getMetrics methods. Let’s start with measureText—it needs to replace all calls to measure. For example, in the draw method for a Tab, we must do:

class Tab:
    def draw(self, canvas):
        if self.focus:
            # ...
            x = obj.x + obj.font.measureText(text)
            # ...

There are also measure calls in DrawText, in the draw method on Browser, in the text method in BlockLayout, and in the layout method in TextLayout. Update all of them to use measureText.

Also, in the layout method of LineLayout and in DrawText we make calls to the metrics method on fonts. In Skia, this method is called getMetrics, and to get the ascent and descent we use

    -font.getMetrics().fAscent

and

    font.getMetrics().fDescent

Note the negative sign when accessing the ascent. In Skia, ascent and descent are positive if they go downward and negative if they go upward, so ascents will normally be negative, the opposite of Tkinter. There’s no analog for the linespace field that Tkinter provides, but you can use descent minus ascent instead.

You should now be able to run the browser again. It should look and behave just as it did in previous chapters, and it’ll probably feel faster, because Skia and SDL are faster than Tkinter. This is one advantage of Skia: since it is also used by the Chromium browser, we know it has fast, built-in support for all of the shapes we might need.

Let’s reward ourselves for the big refactor with a simple feature that Skia enables: rounded corners of a rectangle via the border-radius CSS property, like this:

<div style="border-radius: 10px; background: lightblue">
    This is some example text.
</div>

Which looks like this:If you’re very observant, you may notice that the text here protrudes past the background by just a handful of pixels. This is the correct default behavior, and can be modified by the overflow CSS property, which we’ll see later this chapter.

Implementing border-radius requires drawing a rounded rectangle, so let’s add a new DrawRRect command:

class DrawRRect:
    def __init__(self, rect, radius, color):
        self.rect = rect
        self.rrect = skia.RRect.MakeRectXY(rect, radius, radius)
        self.color = color

    def execute(self, scroll, canvas):
        sk_color = parse_color(self.color)
        canvas.drawRRect(self.rrect,
            paint=skia.Paint(Color=sk_color))

Note that Skia supports RRects, or rounded rectangles, natively, so we can just draw one right to a canvas. Now we can draw these rounded rectangles for the background:

class BlockLayout:
    def paint(self, display_list):
        if not is_atomic:
            if bgcolor != "transparent":
                radius = float(
                    self.node.style.get("border-radius", "0px")[:-2])
                display_list.append(DrawRRect(rect, radius, bgcolor))

Similar changes should be made to InputLayout.

Pixels, color, and raster

Skia, like the Tkinter canvas we’ve been using until now, is a rasterization library: it converts shapes like rectangles and text into pixels. Before we move on to Skia’s advanced features, let’s talk about how rasterization works at a deeper level. This will help to understand how exactly those features work.

You probably already know that computer screens are a 2D array of pixels. Each pixel contains red, green and blue lights,Actually, some screens contain pixels besides red, green, and blue, including white, cyan, or yellow. Moreover, different screens can use slightly different reds, greens, or blues; professional color designers typically have to calibrate their screen to display colors accurately. For the rest of us, the software still communicates with the display in terms of standard red, green, and blue colors, and the display hardware converts to whatever pixels it uses. or color channels, that can shine with an intensity between 0 (off) and 1 (fully on). By mixing red, green, and blue, which is formally known as the sRGB color space, any color in that space’s gamut can be made.The sRGB color space dates back to CRT displays. New technologies like LCD, LED, and OLED can display more colors, so CSS now includes syntax for expressing these new colors. All color spaces have a limited gamut of expressible colors. In a rasterization library, a 2D array of pixels like this is called a surface.Sometimes they are called bitmaps or textures as well, but these words connote specific CPU or GPU technologies for implementing surfaces. Since modern devices have lots of pixels, surfaces require a lot of memory, and we’ll typically want to create as few as possible.

The job of a rasterization library is to determine the red, green, and blue intensity of each pixel on the screen, based on the shapes—lines, rectangles, text—that the application wants to display. The interface for drawing shapes onto a surface is called a canvas; both Tkinter and Skia had canvas APIs. In Skia, each surface has an associated canvas that draws to that surface.

Blending and stacking

Drawing shapes quickly is already a challenge, but with multiple shapes there’s an additional question: what color should the pixel be when two shapes overlap? So far, our browser has only handled opaque shapes,It also hasn’t considered subpixel geometry or anti-aliasing, which also rely on color mixing. and the answer has been simple: take the color of the top shape. But now we need more nuance.

Many objects in nature are partially transparent: frosted glass, clouds, or colored paper, for example. Looking through one, you see multiple colors blended together. That’s also why computer screens work: the red, green, and blue lights blend together and appear to our eyes as another color. Designers use this effectMostly. Some more advanced blending modes on the web are difficult, or perhaps impossible, in real-world physics. in overlays, shadows, and tooltips, so our browser needs to support color mixing.

Color mixing means we need to think carefully about the order of operations. For example, consider black text on an orange background, placed semi-transparently over a white background. The text is gray while the background is yellow-orange. That’s due to blending: the text and the background are both partially transparent and let through some of the underlying white:

But importantly, the text isn’t orange-gray: even though the text is partially transparent, none of the orange shines through. That’s because the order matters: the text is first blended with the background; since the text is opaque, its blended pixels are black and overwrite the orange background. Only then is this black-and-orange image blended with the white background. Doing the operations in a different order would lead to dark-orange or black text.

To handle this properly, browsers apply blending not to individual shapes but to a tree of stacking contexts. Conceptually, each stacking context is drawn onto its own surface, and then blended into its parent stacking context. Rastering a web page requires a bottom-up traversal of the tree of stacking contexts: to raster a stacking context you first need to raster its contents, including its child stacking contexts, and then the whole contents need to be blended together into the parent.

To match this use pattern, in Skia, surfaces form a stack. You can push a new surface on the stack, raster things to it, and then pop it off by blending it with surface below. When traversing the tree of stacking contexts, you push a new surface onto the stack every time you recurse into a new stacking context, and pop-and-blend every time you return from a child stacking context to its parent.

In real browsers, stacking contexts are formed by HTML elements with certain styles, up to any descendants that themselves have such styles. The full definition is actually quite complicated, so in this chapter we’ll simplify by treating every layout object as a stacking context.

Opacity and alpha

Color mixing happens when multiple page elements overlap. The easiest way that happens in our browser is child elements overlapping their parents, like this:There are many more ways elements can overlap in a real browser: the transform property, positioned elements, negative margins, and so many more. But color mixing works the same way each time.

<div style="background-color:orange">
    Parent
    <div style="background-color:white;border-radius:5px">Child</div>
    Parent
</div>

It looks like this:

Right now, the white rectangle completely obscures part of the orange one; the two colors don’t really need to “mix”, and in fact it kind of looks like two orange rectangles instead of an orange rectangle with a white one on top. Now let’s make the white child element semi-transparent, so the colors have to mix. In CSS, that requires adding an opacity property with a value somewhere between 0 (completely transparent) and 1 (totally opaque). With 50% opacity on the white child element, it looks like this:

Notice that instead of being pure white, the child element now has a light-orange background color, resulting from orange and white mixing. Let’s implement this in our browser.

The way to mix colors in Skia is to first create two surfaces, and then draw one into the other. The most convenient way to do that is with saveLayerIt’s called saveLayer instead of createSurface because Skia doesn’t actually promise to create a new surface, if it can optimize that away. So what you’re really doing with saveLayer is telling Skia that there is a new conceptual layer (“piece of paper”) on the stack. Skia’s terminology distinguishes between a layer and a surface for this reason as well, but for our purposes it makes sense to assume that each new layer comes with a surface. and restore:

# draw parent
canvas.saveLayer(paint=skia.Paint(Alphaf=0.5))
# draw child
canvas.restore()

We first draw the parent, then create a new surface with saveLayer to draw the child into, and then when the restore call is made the paint parameters passed into saveLayer are used to mix the colors in the two surfaces together. Here we’re using the Alphaf parameter, which describes the opacity as a floating-point number from 0 to 1.

Note that saveLayer and restore are like a pair of parentheses enclosing the child drawing operations. This means our display list is no longer just a linear sequence of drawing operations, but a tree. So in our display list, let’s represent saveLayer with a SaveLayer command that takes a sequence of other drawing commands as an argument:

class SaveLayer:
    def __init__(self, sk_paint, children):
        self.sk_paint = sk_paint
        self.children = children
        self.rect = skia.Rect.MakeEmpty()
        for cmd in self.children:
            self.rect.join(cmd.rect)

    def execute(self, scroll, canvas):
        canvas.saveLayer(paint=self.sk_paint)
        for cmd in self.children:
            cmd.execute(scroll, canvas)
        canvas.restore()

Now let’s look at how we can add this to our existing paint method for BlockLayouts. Right now, this method draws a background and then recurses into its children, adding each drawing command straight to the global display list. Let’s instead add those drawing commands to a temporary list first:

class BlockLayout:
    def paint(self, display_list):
        cmds = []
        # ...
        if bgcolor != "transparent":
            # ...
            cmds.append(DrawRRect(rect, radius, bgcolor))

        for child in self.children:
            child.paint(cmds)
        # ...        
        display_list.extend(cmds)

Now, before we add our temporary command list to the overall display list, we can use SaveLayer to add transparency to the whole element. I’m going to do this in a new paint_visual_effects method, because we’ll want to make the same changes to all of our other layout objects:

class BlockLayout:
    def paint(self, display_list):
        # ...
        cmds = paint_visual_effects(self.node, cmds, rect)
        display_list.extend(cmds)

Inside paint_visual_effects, we’ll parse the opacity value and construct the appropriate SaveLayer:

def paint_visual_effects(node, cmds, rect):
    opacity = float(node.style.get("opacity", "1.0"))

    return [
        SaveLayer(skia.Paint(Alphaf=opacity), cmds)
    ]

Note that paint_visual_effects receives a list of commands and returns another list of commands. It’s just that the output list is always a single SaveLayer command that wraps the original content—which makes sense, because first we need to draw the commands to a surface, and then apply transparency to it when blending into the parent.

Compositing pixels

Now let’s pause and explore how opacity actually works under the hood. Skia, SDL, and many other color libraries account for opacity with a fourth alpha value for each pixel.The difference between opacity and alpha can be confusing. Think of opacity as a visual effect applied to content, but alpha as a part of content. Think of alpha as implementation technique for representing opacity. An alpha of 0 means the pixel is fully transparent (meaning, no matter what the colors are, you can’t see them anyway), and an alpha of 1 means a fully opaque.

When a pixel with alpha overlaps another pixel, the final color is a mix of their two colors. How exactly the colors are mixed is defined by Skia’s Paint objects. Of course, Skia is pretty complex, but we can sketch these paint operations in Python as methods on an imaginary Pixel class.

class Pixel:
    def __init__(self, r, g, b, a):
        self.r = r
        self.g = g
        self.b = b
        self.a = a

When we apply a Paint with an Alphaf parameter, the first thing Skia does is add the requested opacity to each pixel:

class Pixel:
    def alphaf(self, opacity):
        self.a = self.a * opacity

I want to emphasize that this code is not a part of our browser—I’m simply using Python code to illustrate what Skia is doing internally.

That Alphaf operation applies to pixels in one surface. But with SaveLayer we will end up with two surfaces, with all of their pixels aligned, and therefore we will need to combine, or blend, corresponding pairs of pixels.

Here the terminology can get confusing: we imagine that the pixels “on top” are blending into the pixels “below”, so we call the top surface the source surface, with source pixels, and the bottom surface the destination surface, with destination pixels. When we combine them, there are lots of ways we could do it, but the default on the web is called “simple alpha compositing” or source-over compositing. In Python, the code to implement it looks like this:The formula for this code can be found here. Note that that page refers to premultiplied alpha colors, but Skia’s API does not use premultiplied representations, and the code below doesn’t either.

class Pixel:
    def source_over(self, source):
        self.a = 1 - (1 - source.a) * (1 - self.a)
        if self.a == 0: return self
        self.r = \
            (self.r * (1 - source.a) * self.a + \
                source.r * source.a) / self.a
        self.g = \
            (self.g * (1 - source.a) * self.a + \
                source.g * source.a) / self.a
        self.b = \
            (self.b * (1 - source.a) * self.a + \
                source.b * source.a) / self.a

Here the destination pixel self is modified to blend in the source pixel source. The mathematical expressions for the red, green, and blue color channels are identical, and basically average the source and destination colors, weighted by alpha.For example, if the alpha of the source pixel is 1, the result is just the source pixel color, and if it is 0 the result is the backdrop pixel color. You might imagine the overall operation of SaveLayer with an Alphaf parameter as something like this:In reality, reading individual pixels into memory to manipulate them like this is slow. So libraries such as Skia don’t make it convenient to do so. (Skia canvases do have peekPixels and readPixels methods that are sometimes used, but not for this.)

for (x, y) in destination.coordinates():
    source[x, y].alphaf(opacity)
    destination[x, y].source_over(source[x, y])

Source-over compositing is one way to combine two pixel values. But it’s not the only method—you could write literally any computation that combines two pixel values if you wanted. Two computations that produce interesting effects are traditionally called “multiply” and “difference” and use simple mathematical operations. “Multiply” multiplies the color values:

class Pixel:
    def multiply(self, source):
        self.r = self.r * source.r
        self.g = self.g * source.g
        self.b = self.b * source.b

And “difference” computes their absolute differences:

class Pixel:
    def difference(self, source):
        self.r = abs(self.r - source.r)
        self.g = abs(self.g - source.g)
        self.b = abs(self.b - source.b)

CSS supports these and many other blending modesMany of these blending modes are common to other graphics editing programs like Photoshop and GIMP. Some, like “dodge” and “burn”, go back to analog photography, where photographers would expose some parts of the image more than others to manipulate their brightness. via the mix-blend-mode property, like this:

<div style="background-color:orange">
    Parent
    <div style="background-color:blue;mix-blend-mode:difference">
        Child
    </div>
    Parent
</div>

This HTML will look like:

Here, when blue overlaps with orange, we see pink: blue has (red, green, blue) color channels of (0, 0, 1), and orange has (1, .65, 0), so with “difference” blending the resulting pixel will be (1, 0.65, 1), which is pink. On a pixel level, what’s happening is something like this:

for (x, y) in destination.coordinates():
    source[x, y].alphaf(opacity)
    source[x, y].difference(destination[x, y])
    destination[x, y].source_over(source[x, y])

This looks weird, but conceptually it blends the destination into the source (which ignores alpha) and then draws the source over the destination (with alpha considered). In some sense, blending thus happens twice.

Skia supports the multiply and difference blend modes natively:

def parse_blend_mode(blend_mode_str):
    if blend_mode_str == "multiply":
        return skia.BlendMode.kMultiply
    elif blend_mode_str == "difference":
        return skia.BlendMode.kDifference
    else:
        return skia.BlendMode.kSrcOver

This makes adding support for blend modes to our browser as simple as passing the BlendMode parameter to the Paint object:

def paint_visual_effects(node, cmds, rect):
    # ...
    blend_mode = parse_blend_mode(node.style.get("mix-blend-mode"))
    
    return [
        SaveLayer(skia.Paint(BlendMode=blend_mode), [
            SaveLayer(skia.Paint(Alphaf=opacity), cmds),
        ]),
    ]

Note the order of operations here: we first apply transparency, and then blend the result into the rest of the page. If we switched the two SaveLayer calls, so that we first applied blending, there wouldn’t be anything to blend it into!

Clipping and masking

The “multiply” and “difference” blend modes can seem kind of obscure, but blend modes are a flexible way to implement per-pixel operations. One common use case is clipping—intersecting a surface with a given shape. It’s called clipping because it’s like putting a second piece of paper (called a mask) over the first one, and then using scissors to cut along the mask’s edge.

There are all sorts of powerful methodsThe CSS clip-path property lets specify a mask shape using a curve, while the mask property lets you instead specify a image URL for the mask. for clipping content on the web, but the most common form involves the overflow property. This property has lots of possible values,For example, overflow: scroll adds scroll bars and makes an element scrollable, while overflow: hidden is similar to but subtly different from overflow: clip. but let’s focus here on overflow: clip, which cuts off contents of an element that are outside the element’s bounds.

Usually, overflow: clip is used with properties like height or rotate which can make an element’s children poke outside their parent. Our browser doesn’t support these, but there is one edge case where overflow: clip is relevant: rounded corners. Consider this example:

<div 
  style="border-radius:30px;background-color:lightblue;overflow:clip">
    This test text exists here to ensure that the "div" element is
    large enough that the border radius is obvious.
</div>

That HTML looks like this:

Observe that the letters near the corner are cut off to maintain a sharp rounded edge. (Uhh… actually, at the time of this writing, Safari does not support overflow: clip, so if you’re using Safari you won’t see this effect.The similar overflow: hidden is supported by all browsers. However, in this case, overflow: hidden will also increase the height of div until the rounded corners no longer clip out the text. This is because overflow:hidden has different rules for sizing boxes, having to do with the possibility of the child content being scrolled—hidden means “clipped, but might be scrolled by JavaScript”. If the blue box had not been taller, than it would have been impossible to see the text, which is really bad if it’s intended that there should be a way to scroll it on-screen.) That’s clipping; without the overflow: clip property these letters would instead be fully drawn, like we saw earlier in this chapter.

Counterintuitively, we’ll implement clipping using blending modes. We’ll make a new surface (the mask), draw a rounded rectangle into it, and then blend it with the element contents. But we want to see the element contents, not the mask, so when we do this blending we will use destination-in compositing.

Destination-in compositing basically means keeping the pixels of the destination surface that intersect with the source surface. The source surface’s color is not used—just its alpha. In our case, the source surface is the rounded rectangle mask and the destination surface is the content we want to clip, so destination-in fits perfectly. In code, destination-in looks like this:

class Pixel:
    def destination_in(self, source):
        self.a = self.a * source.a
        if self.a == 0: return self
        self.r = (self.r * self.a * source.a) / self.a
        self.g = (self.g * self.a * source.a) / self.a
        self.b = (self.b * self.a * source.a) / self.a

Now, in paint_visual_effects, we need to create a new layer, draw the mask image into it, and then blend it with the element contents with destination-in blending:

def paint_visual_effects(node, cmds, rect):
    # ...
    border_radius = float(node.style.get("border-radius", "0px")[:-2])
    if node.style.get("overflow", "visible") == "clip":
        clip_radius = border_radius
    else:
        clip_radius = 0


    return [
        SaveLayer(skia.Paint(BlendMode=blend_mode), [
            SaveLayer(skia.Paint(Alphaf=opacity), cmds),
            SaveLayer(skia.Paint(BlendMode=skia.kDstIn), [
                DrawRRect(rect, clip_radius, "white")
            ]),
        ]),
    ]

After drawing all of the element contents with cmds (and applying opacity), this code draws a rounded rectangle on another layer to serve as the mask, and uses destination-in blending to clip the element contents. Here I chose to draw the rounded rectangle in white, but the color doesn’t matter as long as it’s opaque. On the other hand, if there’s no clipping, I don’t round the corners of the mask, which means nothing is clipped out.

Notice how similar this masking technique is to the physical analogy with scissors described earlier, with the two layers playing the role of two sheets of paper and destination-in compositing playing the role of the scissors. This implementation technique for clipping is called masking, and it is very general—you can use it with arbitrarily complex mask shapes, like text, bitmap images, or anything else you can imagine.

Optimizing surface use

Our browser now works correctly, but uses way too many surfaces. For example, for a single, no-effects-needed div with some text content, there are currently 18 surfaces allocated in the display list. If there’s no blending going on, we should only need one!

Let’s review all the surfaces that our code can create for an element:

• The top-level surface is used to apply blend modes. Since it’s the top-level surface, it also isolates the element from other parts of the page, so that clipping only applies to that element.
• The first nested surface is used for applying opacity.
• The second nested surface is used to implement clipping.

But not every element has opacity, blend modes, or clipping applied, and we could skip creating those surfaces most of the time. To implement this without making the code hard to read, let’s change SaveLayer to take two additional optional parameters: should_save and should_paint_cmds. These control whether saveLayer is called and whether subcommands are actually painted:

class SaveLayer:
    def __init__(self, sk_paint, children,
            should_save=True, should_paint_cmds=True):
        self.should_save = should_save
        self.should_paint_cmds = should_paint_cmds
        # ...

    def execute(self, canvas):
        if self.should_save:
            canvas.saveLayer(paint=self.sk_paint)
        if self.should_paint_cmds:
            for cmd in self.children:
                cmd.execute(canvas)
        if self.should_save:
            canvas.restore()

Turn off those parameters if an effect isn’t applied:

def paint_visual_effects(node, cmds, rect):
    # ...

    needs_clip = node.style.get("overflow", "visible") == "clip"
    needs_blend_isolation = blend_mode != skia.BlendMode.kSrcOver or \
        needs_clip
    needs_opacity = opacity != 1.0

   return [
        SaveLayer(skia.Paint(BlendMode=blend_mode), [
            SaveLayer(skia.Paint(Alphaf=opacity), cmds,
                should_save=needs_opacity),
            SaveLayer(skia.Paint(BlendMode=skia.kDstIn), [
                DrawRRect(rect, clip_radius, "white")
            ], should_save=needs_clip, should_paint_cmds=needs_clip),
        ], should_save=needs_blend_isolation),
    ]

Now simple web pages always use a single surface—a huge saving in memory. But we can save even more surfaces. For example, what if there is a blend mode and opacity at the same time: can we use the same surface? Indeed, yes you can! That’s also pretty simple:This works for opacity, but not for filters that “move pixels” such as blur. Such a filter needs to be applied before clipping, not when blending into the parent surface. Otherwise, the edge of the blur will not be sharp.

def paint_visual_effects(node, cmds, rect):
    # ...

    needs_clip = node.style.get("overflow", "visible") == "clip"
    needs_blend_isolation = blend_mode != skia.BlendMode.kSrcOver or \
        needs_clip
    needs_opacity = opacity != 1.0

   return [
        SaveLayer(skia.Paint(BlendMode=blend_mode, Alphaf=opacity),
            cmds + [
            SaveLayer(skia.Paint(BlendMode=skia.kDstIn), [
                DrawRRect(rect, clip_radius, "white")
            ], should_save=needs_clip, should_paint_cmds=needs_clip),
        ], should_save=needs_blend_isolation or needs_opacity),
    ]

There’s one more optimization to make: using Skia’s clipRRect operation to get rid of the destination-in blended surface. This operation takes in a rounded rectangle and changes the canvas state so that all future commands skip drawing any pixels outside that rounded rectangle.

There are multiple advantages to using clipRRect over an explicit destination-in surface. First, most of the time, it allows Skia to avoid making a surface for the mask.Typically in a browser this means code in GPU shaders. GPU programs are out of scope for this book, but if you’re curious there are many online resources describing ways to do this. It also allows Skia to skip draw operations that don’t intersect the mask, or dynamically draw only the parts of operations that intersect it. It’s basically the optimization we implemented for scrolling in Chapter 2.This kind of code is complex for Skia to implement, so it only makes sense to do it for common patterns, like rounded rectangles. This is why Skia only supports optimized clips for a few common shapes.

Since clipRRect changes the canvas state, we’ll need to restore it once we’re done with clipping. That uses the save and restore methods—you call save before calling clipRRect, and restore after finishing drawing the commands that should be clipped:

# Draw commands that should not be clipped.
canvas.save()
canvas.clipRRect(rounded_rect)
# Draw commands that should be clipped.
canvas.restore()
# Draw commands that should not be clipped.

If you’ve noticed that restore is used for both saving state and pushing surfaces, that’s because Skia has a combined stack of surfaces and canvas states. Unlike saveLayer, however, save never creates a new surface.

Let’s wrap this pattern into a ClipRRect drawing command, which like SaveLayer takes a list of subcommands and a should_clip parameter indicating whether the clip is necessary:If you’re doing two clips at once, or a clip and a transform, or some other more complex setup that would benefit from only saving once but doing multiple things inside it, this pattern of always saving canvas parameters might be wasteful, but since it doesn’t create a surface it’s still a big optimization here.

class ClipRRect:
    def __init__(self, rect, radius, children, should_clip=True):
        self.rect = rect
        self.rrect = skia.RRect.MakeRectXY(rect, radius, radius)
        self.children = children
        self.should_clip = should_clip

    def execute(self, canvas):
        if self.should_clip:
            canvas.save()
            canvas.clipRRect(self.rrect)

        for cmd in self.children:
            cmd.execute(canvas)

        if self.should_clip:
            canvas.restore()

Now, in paint_visual_effects, we can use ClipRRect instead of destination-in blending with DrawRRect (and we can fold the opacity into the skia.Paint passed to the outer SaveLayer, since that is defined to be applied before blending):

def paint_visual_effects(node, cmds, rect):
    # ...
    return [
        SaveLayer(skia.Paint(BlendMode=blend_mode, Alphaf=opacity), [
            ClipRRect(rect, clip_radius,
                cmds,
            should_clip=needs_clip),
        ], should_save=needs_blend_isolation),
    ]

Of course, clipRRect only applies for rounded rectangles, while masking is a general technique that can be used to implement all sorts of clips and masks (like CSS’s clip-path and mask), so a real browser will typically have both code paths.

So now, each element uses at most one surface, and even then only if it has opacity or a non-default blend mode. Everything else should look visually the same, but will be faster and use less memory.

Browser compositing

Optimizing away surfaces is great when they’re not needed, but sometimes having more surfaces allows faster scrolling and animations.

So far, any time anything changed in the browser chrome or the web page itself, we had to clear the canvas and re-raster everything on it from scratch. This is inefficient—ideally, things should be re-rastered only if they actually change. When the context is complex or the screen is large, rastering too often produces a visible slowdown, and laptop and mobile batteries are drained unnecessarily. Real browsers optimize these situations by using a technique I’ll call browser compositing. The idea is to create a tree of explicitly cached surfaces for different pieces of content. Whenever something changes, we’ll re-raster only the surface where that content appears. Then these surfaces are blended (or “composited”) together to form the final image that the user sees.

Let’s implement this, with a surface for browser chrome and a surface for the current Tab’s contents. This way, we’ll only need to re-raster the Tab surface if page contents change, but not when (say) the user types into the address bar. This technique also allows us to scroll the Tab without any raster at all—we can just translate the page contents surface when drawing it.

To start with, we’ll need two new surfaces on Browser, chrome_surface and tab_surface:We could even use a different surface for each Tab, but real browsers don’t do this, since each surface uses up a lot of memory, and typically users don’t notice the small raster delay when switching tabs.

class Browser:
    def __init__(self):
        # ...
        self.chrome_surface = skia.Surface(WIDTH, CHROME_PX)
        self.tab_surface = None

I’m not explicitly creating tab_surface right away, because we need to lay out the page contents to know how tall the surface needs to be.

We’ll also need to split the browser’s draw method into three parts:

• draw will composite the chrome and tab surfaces and copy the result from Skia to SDL;
• raster_tab will draw the page to the tab_surface; and
• raster_chrome will draw the browser chrome to the chrome_surface.

Let’s start by doing the split:

class Browser:
    def raster_tab(self):
        canvas = self.tab_surface.getCanvas()
        canvas.clear(skia.ColorWHITE)
        # ...

    def raster_chrome(self):
        canvas = self.chrome_surface.getCanvas()
        canvas.clear(skia.ColorWHITE)
        # ...

    def draw(self):
        canvas = self.root_surface.getCanvas()
        canvas.clear(skia.ColorWHITE)
        # ...

Since we didn’t create the tab_surface on startup, we need to create it at the top of raster_tab:For a very big web page, the tab_surface can be much larger than the size of the SDL window, and therefore take up a very large amount of memory. We’ll ignore that, but a real browser would only paint and raster surface content up to a certain distance from the visible region, and re-paint/raster as the user scrolls.

import math

class Browser:
    def raster_tab(self):
        active_tab = self.tabs[self.active_tab]
        tab_height = math.ceil(active_tab.document.height)

        if not self.tab_surface or \
                tab_height != self.tab_surface.height():
            self.tab_surface = skia.Surface(WIDTH, tab_height)

        # ...

The way we compute the page bounds here, based on the layout tree’s height, would be incorrect if page elements could stick out below (or to the right) of their parents—but our browser doesn’t support any features like that. Note that we need to recreate the tab surface if the page’s height changes.

Next, we need new code in draw to copy from the chrome and tab surfaces to the root surface. Moreover, we need to translate the tab_surface down by CHROME_PX and up by scroll, and clips it to only the area of the window that doesn’t overlap the browser chrome:

class Browser:
    def draw(self):
        # ...
        
        tab_rect = skia.Rect.MakeLTRB(0, CHROME_PX, WIDTH, HEIGHT)
        tab_offset = CHROME_PX - self.tabs[self.active_tab].scroll
        canvas.save()
        canvas.clipRect(tab_rect)
        canvas.translate(0, tab_offset)
        self.tab_surface.draw(canvas, 0, 0)
        canvas.restore()

        chrome_rect = skia.Rect.MakeLTRB(0, 0, WIDTH, CHROME_PX)
        canvas.save()
        canvas.clipRect(chrome_rect)
        self.chrome_surface.draw(canvas, 0, 0)
        canvas.restore()

        # ...

Finally, everywhere in Browser that we call draw, we now need to call either raster_tab or raster_chrome first. For example, in handle_click, we do this:

class Browser:
    def handle_click(self, e):
        if e.y < CHROME_PX:
            # ...
            self.raster_chrome()
        else:
            # ...
            self.raster_tab()
        self.draw()

Notice how we don’t redraw the chrome when only the tab changes, and vice versa. In handle_down, which scrolls the page, we don’t need to call raster_tab at all, since scrolling doesn’t change the page.

We also have some related changes in Tab. First, we no longer need to pass around the scroll offset to the execute methods, or account for CHROME_PX, because we always draw the whole tab to the tab surface:

class Tab:
    def raster(self, canvas):
        for cmd in self.display_list:
            cmd.execute(canvas)

Likewise, we can remove the scroll parameter from each drawing command’s execute method:

class DrawRect:
    def execute(self, canvas):
        draw_rect(canvas,
            self.left, self.top,
            self.right, self.bottom,
            fill=self.color, width=0)

Our browser now uses composited scrolling, making scrolling faster and smoother. In fact, in terms of conceptual phases of execution, our browser is now very close to real browsers: real browsers paint display lists, break content up into different rastered surfaces, and finally draw the tree of surfaces to the screen. There’s more we can do for performance—ideally we’d avoid all duplicate or unnecessary operations—but let’s leave that for the next few chapters.

Summary

So there you have it: our browser can draw not only boring text and boxes but also:

• Partial transparency via an alpha channel
• User-configurable blending modes via mix-blend-mode
• Rounded rectangle clipping via destination-in blending or direct clipping
• Optimizations to avoid surfaces when possible
• Browser compositing with extra surfaces for faster scrolling

Besides the new features, we’ve upgraded from Tkinter to SDL and Skia, which makes our browser faster and more responsive, and also sets a foundation for more work on browser performance to come.

Outline

The complete set of functions, classes, and methods in our browser should now look something like this:

Exercises

CSS transforms: Add support for the transform CSS property, specifically the translate and rotate transforms.There is a lot more complexity to 3D transforms having to do with the definition of 3D spaces, flatting, backfaces, and plane intersections. Skia has built-in support for these via canvas state.

Filters: The filter CSS property allows specifying various kinds of more complex effects, such as grayscale or blur. These are fun to implement, and a number of them have built-in support in Skia. Implement, for example, the blur filter. Think carefully about when filters occur, relative to other effects like transparency, clipping, and blending.

Hit testing: If you have an element with a border-radius, it’s possible to click outside the element but inside its containing rectangle, by clicking in the part of the corner that is “rounded off”. This shouldn’t result in clicking on the element, but in our browser it currently does. Modify the click method to take border radii into account.

Interest region: Our browser now draws the whole web page to a single surface, and then shows parts of that surface as the user scrolls. That means a very long web page (like this one!) can create a large surface, thereby using a lot of memory. Modify the browser so that the height of that surface is limited, say to 4 * HEIGHT pixels. The (limited) region of the page drawn to this surface is called the interest region; you’ll need to track what part of the interest region is being shown on the screen, and re-raster the interest region when the user attempts to scroll outside of it.

One way to do this is to filter out all display list items that don’t intersect the interest rect. Another, easier way is to take advantage of Skia’s internal optimizations: if you call save and clipRect on a Skia canvas and then some draw operations, Skia will automatically avoid display item raster work outside of the clipping rectangle before the next restore.

Z-index: Right now, elements later in the HTML document are drawn “on top” of earlier ones. The z-index CSS property changes that order: an element with the larger z-index draws on top (with ties broken by the current order, and with the default z-index being 0). For z-index to have any effect, the element’s position property must be set to something other than static (the default). Add support for z-index. One thing you’ll run into is that with our browser’s minimal layout features, you might not be able to create any overlapping elements to test this feature! However, lots of exercises throughout the book allow you to create overlapping elements, including transform and width/height. For an extra challenge, add support for nested elements with z-index properties.

Overflow scrolling: An element with the overflow property set to scroll and a fixed pixel height is scrollable. (You’ll want to implement the width/height exercise from Chapter 6 so that height is supported.) Implement some version of overflow: scroll. I recommend the following user interaction: the user clicks within a scrollable element to focus it, and then can press the arrow keys to scroll up and down. You’ll need to keep track of the layout overflow. For an extra challenge, make sure you support scrollable elements nested within other scrollable elements.

Did you find this chapter useful? Send us feedback!