Adding Visual Effects | Web Browser Engineering

Right now our browser can only draw colored rectangles and text—pretty boring! Real browsers support all kinds of visual effects that change how pixels and colors blend together. Let’s implement these effects using the Skia graphics library, and also see a bit of how Skia is implemented under the hood. That’ll also allow us to use surfaces for browser compositing to accelerate scrolling.

Installing Skia and SDL

Before we get any further, we’ll need to upgrade our graphics system. While Tkinter is great for basic shapes and handling input, it lacks built-in support for many visual effects.That’s because Tk, the graphics library that Tkinter uses, dates from the early 90s, before high-performance graphics cards and GPUs became widespread. Implementing all details of the web’s many visual effects is fun, but it’s outside the scope of this book, so we need a new graphics library. Let’s use Skia, the library that Chromium uses. Unlike Tkinter, Skia doesn’t handle inputs or create graphical windows, so we’ll pair it with the SDL GUI library.

Once installed, remove the tkinter imports from browser and replace them with these:

If any of these imports fail, you probably need to check that Skia and SDL were installed correctly. Note that the ctypes module comes standard in Python; it is used to convert between Python and C types.

SDL creates the window

Next, we need to create an SDL window, instead of a Tkinter window, inside the Browser, and set up Skia to draw to it. Here’s the SDL incantation to create a window:

To set up Skia to draw to this window, we also need to create a surface for it:In Skia and SDL, a surface is a representation of a graphics buffer into which you can draw pixels (bits representing colors). A surface may or may not be bound to the physical pixels on the screen via a window, and there can be many surfaces. A canvas is an API interface that allows you to draw into a surface with higher-level commands such as for rectangles or text. Our browser uses separate Skia and SDL surfaces for simplicity, but in a highly optimized browser, minimizing the number of surfaces is important for good performance.

Typically, we’ll draw to the Skia surface, and then once we’re done with it we’ll copy it to the SDL surface to display on the screen. This will be a little hairy, because we are moving data between two low-level libraries, but really it’s just copying pixels from one place to another.

Next, we need to copy the data to an SDL surface. This requires telling SDL what order the pixels are stored in (which we specified to be RGBA_8888 when constructing the surface) and on your computer’s endianness:

The CreateRGBSurfaceFrom method then wraps the data in an SDL surface (this SDL surface does not copy the bytes): Note that since Skia and SDL are C++ libraries, they are not always consistent with Python’s garbage collection system. So the link between the output of tobytes and sdl_window is not guaranteed to be kept consistent when skia_bytes is garbage collected. Instead, the SDL surface will be pointing at a bogus piece of memory, which will lead to memory corruption or a crash. The code here is correct because all of these are local variables that are garbage-collected together, but if not you need to be careful to keep all of them alive at the same time.

Finally, we draw all this pixel data on the window itself by blitting (copying) it from sdl_surface to sdl_window’s surface:

Next, SDL doesn’t have a mainloop or bind method; we have to implement it ourselves:

The details of ctypes and PollEvent aren’t too important here, but note that SDL_QUIT is an event, sent when the user closes the last open window. The handle_quit method it calls just cleans up the window object:

We’ll also need to handle all of the other events in this loop—clicks, typing, and so on. The SDL syntax looks like this:

I’ve changed the signatures of the various event handler methods; you’ll need to make analogous changes in Browser where they are defined. This loop replaces all of the bind calls in the Browser constructor, which you can now remove.

Skia provides the canvas

Now our browser is creating an SDL window and can draw to it via Skia. But most of the browser codebase is still using Tkinter drawing commands, which we now need to replace. Skia is a bit more verbose than Tkinter, so let’s abstract over some details with helper functions.Consult the Skia and skia-python documentation for more on the Skia API. First, a helper function to convert colors to Skia colors:

You can add more “elif” blocks to support any other color names you use; modern browsers support quite a lot.

Finally, for drawing rectangles you use drawRect. Filling in the rectangle is the default:

Here the rect field is a Skia Rect object, which you can construct using MakeLTRB (for “make left-top-right-bottom”) or MakeXYWH (for “make x-y-width-height”):

To draw just the outline, set the Style parameter of the Paint to Stroke_Style. Here “stroke” is a standard term referring to drawing along the border of some shape; the opposite is “fill”, meaning filling in the interior of the shape:

If you look at the details of these helper methods, you’ll see that they all use a Skia Paint object to describe a shape’s borders and colors. We’ll be seeing a lot more features of Paint in this chapter.

While we’re here, let’s also add a rect field to the other drawing commands, replacing its top, left, bottom, and right fields:You’ll probably need some font changes in the browser UI, because Skia draws fonts a bit differently from Tkinter. I had to adjust the y position of the plus sign and less than signs to keep them centered in their boxes. Feel free to adjust to make everything look good on your system.

Since we’re replacing Tkinter with Skia, we are also replacing tkinter.font. In Skia, a font object has two pieces: a Typeface, which is a type family with a certain weight, style, and width; and a Font, which is a Typeface at a particular size. It’s the Typeface that contains data and caches, so that’s what we need to cache:

Our browser also needs font metrics and measurements. In Skia, these are provided by the measureText and getMetrics methods. Let’s start with measureText replacing all calls to measure. For example, in the render method for a Tab, we must do:

There are also measure calls in DrawText, in the draw method on Browser, in the text method in BlockLayout, and in the layout method in TextLayout. Update all of them to use measureText.

Also, in the layout method of LineLayout and in DrawText we make calls to the metrics method on fonts. In Skia, this method is called getMetrics, and to get the ascent and descent we use

Note the negative sign when accessing the ascent. In Skia, ascent and descent are positive if they go downward and negative if they go upward, so ascents will normally be negative, the opposite of Tkinter. There’s no analog for the linespace field that Tkinter provides, but you can use descent minus ascent instead:

You should now be able to run the browser again. It should look and behave just as it did in previous chapters, and it’ll probably feel faster, because Skia and SDL are faster than Tkinter. This is one advantage of Skia: since it is also used by the Chromium browser, we know it has fast, built-in support for all of the shapes we might need. And if the transition felt easy—well, that’s one of the benefits to abstracting over the drawing backend using a display list!

What Skia gives us

Let’s reward ourselves for the big refactor with a simple feature that Skia enables: rounded corners of a rectangle via the border-radius CSS property, like this:

Which looks like this:If you’re very observant, you may notice that the text here protrudes past the background by just a handful of pixels. This is the correct default behavior, and can be modified by the overflow CSS property, which we’ll see later this chapter.

Implementing border-radius requires drawing a rounded rectangle, so let’s add a new DrawRRect command:

Note that Skia supports RRects, or rounded rectangles, natively, so we can just draw one right to a canvas. Now we can draw these rounded rectangles for the background:

Pixels, color, and raster

Skia, like the Tkinter canvas we’ve been using until now, is a rasterization library: it converts shapes like rectangles and text into pixels. Before we move on to Skia’s advanced features, let’s talk about how rasterization works at a deeper level. This will help to understand how exactly those features work.

You probably already know that computer screens are a 2D array of pixels. Each pixel contains red, green and blue lights,Actually, some screens contain pixels besides red, green, and blue, including white, cyan, or yellow. Moreover, different screens can use slightly different reds, greens, or blues; professional color designers typically have to calibrate their screen to display colors accurately. For the rest of us, the software still communicates with the display in terms of standard red, green, and blue colors, and the display hardware converts to whatever pixels it uses. or color channels, that can shine with an intensity between 0 (off) and 1 (fully on). By mixing red, green, and blue, which is formally known as the sRGB color space, any color in that space’s gamut can be made.The sRGB color space dates back to CRT displays. New technologies like LCD, LED, and OLED can display more colors, so CSS now includes syntax for expressing these new colors. All color spaces have a limited gamut of expressible colors. In a rasterization library, a 2D array of pixels like this is called a surface.Sometimes they are called bitmaps or textures as well, but these words connote specific CPU or GPU technologies for implementing surfaces. Since modern devices have lots of pixels, surfaces require a lot of memory, and we’ll typically want to create as few as possible.

The job of a rasterization library is to determine the red, green, and blue intensity of each pixel on the screen, based on the shapes—lines, rectangles, text—that the application wants to display. The interface for drawing shapes onto a surface is called a canvas; both Tkinter and Skia had canvas APIs. In Skia, each surface has an associated canvas that draws to that surface.

Blending and stacking

Drawing shapes quickly is already a challenge, but with multiple shapes there’s an additional question: what color should the pixel be when two shapes overlap? So far, our browser has only handled opaque shapes,It also hasn’t considered subpixel geometry or anti-aliasing, which also rely on color mixing. and the answer has been simple: take the color of the top shape. But now we need more nuance.

Many objects in nature are partially transparent: frosted glass, clouds, or colored paper, for example. Looking through one, you see multiple colors blended together. That’s also why computer screens work: the red, green, and blue lights blend together and appear to our eyes as another color. Designers use this effectMostly. Some more advanced blending modes on the web are difficult, or perhaps impossible, in real-world physics. in overlays, shadows, and tooltips, so our browser needs to support color mixing.

Color mixing means we need to think carefully about the order of operations. For example, consider black text on an orange background, placed semi-transparently over a white background. The text is gray while the background is yellow-orange. That’s due to blending: the text and the background are both partially transparent and let through some of the underlying white:

But importantly, the text isn’t orange-gray: even though the text is partially transparent, none of the orange shines through. That’s because the order matters: the text is first blended with the background; since the text is opaque, its blended pixels are black and overwrite the orange background. Only then is this black-and-orange image blended with the white background. Doing the operations in a different order would lead to dark-orange or black text.

To handle this properly, browsers apply blending not to individual shapes but to a tree of stacking contexts. Conceptually, each stacking context is drawn onto its own surface, and then blended into its parent stacking context. Rastering a web page requires a bottom-up traversal of the tree of stacking contexts: to raster a stacking context you first need to raster its contents, including its child stacking contexts, and then the whole contents need to be blended together into the parent.

To match this use pattern, in Skia, surfaces form a stack. You can push a new surface on the stack, raster things to it, and then pop it off by blending it with surface below. When traversing the tree of stacking contexts, you push a new surface onto the stack every time you recurse into a new stacking context, and pop-and-blend every time you return from a child stacking context to its parent.

In real browsers, stacking contexts are formed by HTML elements with certain styles, up to any descendants that themselves have such styles. The full definition is actually quite complicated, so in this chapter we’ll simplify by treating every layout object as a stacking context.

Opacity and alpha

Color mixing happens when multiple page elements overlap. The easiest way that happens in our browser is child elements overlapping their parents, like this:There are many more ways elements can overlap in a real browser: the transform property, positioned elements, negative margins, and so many more. But color mixing works the same way each time.

Right now, the white rectangle completely obscures part of the orange one; the two colors don’t really need to “mix”, and in fact it kind of looks like two orange rectangles instead of an orange rectangle with a white one on top. Now let’s make the white child element semi-transparent, so the colors have to mix. In CSS, that requires adding an opacity property with a value somewhere between 0 (completely transparent) and 1 (totally opaque). With 50% opacity on the white child element, it looks like this:

Notice that instead of being pure white, the child element now has a light-orange background color, resulting from orange and white mixing. Let’s implement this in our browser.

The way to mix colors in Skia is to first create two surfaces, and then draw one into the other. The most convenient way to do that is with saveLayerIt’s called saveLayer instead of createSurface because Skia doesn’t actually promise to create a new surface, if it can optimize that away. So what you’re really doing with saveLayer is telling Skia that there is a new conceptual layer (“piece of paper”) on the stack. Skia’s terminology distinguishes between a layer and a surface for this reason as well, but for our purposes it makes sense to assume that each new layer comes with a surface. and restore:

We first draw the parent, then create a new surface with saveLayer to draw the child into, and then when the restore call is made the paint parameters passed into saveLayer are used to mix the colors in the two surfaces together. Here we’re using the Alphaf parameter, which describes the opacity as a floating-point number from 0 to 1.

Note that saveLayer and restore are like a pair of parentheses enclosing the child drawing operations. This means our display list is no longer just a linear sequence of drawing operations, but a tree. So in our display list, let’s represent saveLayer with a SaveLayer command that takes a sequence of other drawing commands as an argument:

Now let’s look at how we can add this to our existing paint method for BlockLayouts. Right now, this method draws a background and then recurses into its children, adding each drawing command straight to the global display list. Let’s instead add those drawing commands to a temporary list first:

Now, before we add our temporary command list to the overall display list, we can use SaveLayer to add transparency to the whole element. I’m going to do this in a new paint_visual_effects method, because we’ll want to make the same changes to all of our other layout objects:

Inside paint_visual_effects, we’ll parse the opacity value and construct the appropriate SaveLayer:

Note that paint_visual_effects receives a list of commands and returns another list of commands. It’s just that the output list is always a single SaveLayer command that wraps the original content—which makes sense, because first we need to draw the commands to a surface, and then apply transparency to it when blending into the parent.

Compositing pixels

Now let’s pause and explore how opacity actually works under the hood. Skia, SDL, and many other color libraries account for opacity with a fourth alpha value for each pixel.The difference between opacity and alpha can be confusing. Think of opacity as a visual effect applied to content, but alpha as a part of content. Think of alpha as implementation technique for representing opacity. An alpha of 0 means the pixel is fully transparent (meaning, no matter what the colors are, you can’t see them anyway), and an alpha of 1 means a fully opaque.

When a pixel with alpha overlaps another pixel, the final color is a mix of their two colors. How exactly the colors are mixed is defined by Skia’s Paint objects. Of course, Skia is pretty complex, but we can sketch these paint operations in Python as methods on an imaginary Pixel class.

When we apply a Paint with an Alphaf parameter, the first thing Skia does is add the requested opacity to each pixel:

I want to emphasize that this code is not a part of our browser—I’m simply using Python code to illustrate what Skia is doing internally.

That Alphaf operation applies to pixels in one surface. But with SaveLayer we will end up with two surfaces, with all of their pixels aligned, and therefore we will need to combine, or blend, corresponding pairs of pixels.

Here the terminology can get confusing: we imagine that the pixels “on top” are blending into the pixels “below”, so we call the top surface the source surface, with source pixels, and the bottom surface the destination surface, with destination pixels. When we combine them, there are lots of ways we could do it, but the default on the web is called “simple alpha compositing” or source-over compositing. In Python, the code to implement it looks like this:The formula for this code can be found here. Note that that page refers to premultiplied alpha colors, but Skia’s API does not use premultiplied representations, and the code below doesn’t either.

Here the destination pixel self is modified to blend in the source pixel source. The mathematical expressions for the red, green, and blue color channels are identical, and basically average the source and destination colors, weighted by alpha.For example, if the alpha of the source pixel is 1, the result is just the source pixel color, and if it is 0 the result is the backdrop pixel color. You might imagine the overall operation of SaveLayer with an Alphaf parameter as something like this:In reality, reading individual pixels into memory to manipulate them like this is slow. So libraries such as Skia don’t make it convenient to do so. (Skia canvases do have peekPixels and readPixels methods that are sometimes used, but not for this.)

Source-over compositing is one way to combine two pixel values. But it’s not the only method—you could write literally any computation that combines two pixel values if you wanted. Two computations that produce interesting effects are traditionally called “multiply” and “difference” and use simple mathematical operations. “Multiply” multiplies the color values:

CSS supports these and many other blending modesMany of these blending modes are common to other graphics editing programs like Photoshop and GIMP. Some, like “dodge” and “burn”, go back to analog photography, where photographers would expose some parts of the image more than others to manipulate their brightness. via the mix-blend-mode property, like this:

Here, when blue overlaps with orange, we see pink: blue has (red, green, blue) color channels of (0, 0, 1), and orange has (1, .65, 0), so with “difference” blending the resulting pixel will be (1, 0.65, 1), which is pink. On a pixel level, what’s happening is something like this:

This looks weird, but conceptually it blends the destination into the source (which ignores alpha) and then draws the source over the destination (with alpha considered). In some sense, blending thus happens twice.

This makes adding support for blend modes to our browser as simple as passing the BlendMode parameter to the Paint object:

Note the order of operations here: we first apply transparency, and then blend the result into the rest of the page. If we switched the two SaveLayer calls, so that we first applied blending, there wouldn’t be anything to blend it into!

Clipping and masking

The “multiply” and “difference” blend modes can seem kind of obscure, but blend modes are a flexible way to implement per-pixel operations. One common use case is clipping—intersecting a surface with a given shape. It’s called clipping because it’s like putting a second piece of paper (called a mask) over the first one, and then using scissors to cut along the mask’s edge.

There are all sorts of powerful methodsThe CSS clip-path property lets specify a mask shape using a curve, while the mask property lets you instead specify a image URL for the mask. for clipping content on the web, but the most common form involves the overflow property. This property has lots of possible values,For example, overflow: scroll adds scroll bars and makes an element scrollable, while overflow: hidden is similar to but subtly different from overflow: clip. but let’s focus here on overflow: clip, which cuts off contents of an element that are outside the element’s bounds.

Usually, overflow: clip is used with properties like height or rotate which can make an element’s children poke outside their parent. Our browser doesn’t support these, but there is one edge case where overflow: clip is relevant: rounded corners. Consider this example:

Observe that the letters near the corner are cut off to maintain a sharp rounded edge. (Uhh… actually, at the time of this writing, Safari does not support overflow: clip, so if you’re using Safari you won’t see this effect.The similar overflow: hidden is supported by all browsers. However, in this case, overflow: hidden will also increase the height of div until the rounded corners no longer clip out the text. This is because overflow:hidden has different rules for sizing boxes, having to do with the possibility of the child content being scrolled—hidden means “clipped, but might be scrolled by JavaScript”. If the blue box had not been taller, than it would have been impossible to see the text, which is really bad if it’s intended that there should be a way to scroll it on-screen.) That’s clipping; without the overflow: clip property these letters would instead be fully drawn, like we saw earlier in this chapter.

Counterintuitively, we’ll implement clipping using blending modes. We’ll make a new surface (the mask), draw a rounded rectangle into it, and then blend it with the element contents. But we want to see the element contents, not the mask, so when we do this blending we will use destination-in compositing.

Destination-in compositing basically means keeping the pixels of the destination surface that intersect with the source surface. The source surface’s color is not used—just its alpha. In our case, the source surface is the rounded rectangle mask and the destination surface is the content we want to clip, so destination-in fits perfectly. In code, destination-in looks like this:

Now, in paint_visual_effects, we need to create a new layer, draw the mask image into it, and then blend it with the element contents with destination-in blending:

After drawing all of the element contents with cmds (and applying opacity), this code draws a rounded rectangle on another layer to serve as the mask, and uses destination-in blending to clip the element contents. Here I chose to draw the rounded rectangle in white, but the color doesn’t matter as long as it’s opaque. On the other hand, if there’s no clipping, I don’t round the corners of the mask, which means nothing is clipped out.

Notice how similar this masking technique is to the physical analogy with scissors described earlier, with the two layers playing the role of two sheets of paper and destination-in compositing playing the role of the scissors. This implementation technique for clipping is called masking, and it is very general—you can use it with arbitrarily complex mask shapes, like text, bitmap images, or anything else you can imagine.

Optimizing surface use

Our browser now works correctly, but uses way too many surfaces. For example, for a single, no-effects-needed div with some text content, there are currently 18 surfaces allocated in the display list. If there’s no blending going on, we should only need one!

But not every element has opacity, blend modes, or clipping applied, and we could skip creating those surfaces most of the time. To implement this without making the code hard to read, let’s change SaveLayer to take two additional optional parameters: should_save and should_paint_cmds. These control whether saveLayer is called and whether subcommands are actually painted:

Now simple web pages always use a single surface—a huge saving in memory. But we can save even more surfaces. For example, what if there is a blend mode and opacity at the same time: can we use the same surface? Indeed, yes you can! That’s also pretty simple:This works for opacity, but not for filters that “move pixels” such as blur. Such a filter needs to be applied before clipping, not when blending into the parent surface. Otherwise, the edge of the blur will not be sharp.

There’s one more optimization to make: using Skia’s clipRRect operation to get rid of the destination-in blended surface. This operation takes in a rounded rectangle and changes the canvas state so that all future commands skip drawing any pixels outside that rounded rectangle.

There are multiple advantages to using clipRRect over an explicit destination-in surface. First, most of the time, it allows Skia to avoid making a surface for the mask.Typically in a browser this means code in GPU shaders. GPU programs are out of scope for this book, but if you’re curious there are many online resources describing ways to do this. It also allows Skia to skip draw operations that don’t intersect the mask, or dynamically draw only the parts of operations that intersect it. It’s basically the optimization we implemented for scrolling in Chapter 2.This kind of code is complex for Skia to implement, so it only makes sense to do it for common patterns, like rounded rectangles. This is why Skia only supports optimized clips for a few common shapes.

Since clipRRect changes the canvas state, we’ll need to restore it once we’re done with clipping. That uses the save and restore methods—you call save before calling clipRRect, and restore after finishing drawing the commands that should be clipped:

If you’ve noticed that restore is used for both saving state and pushing surfaces, that’s because Skia has a combined stack of surfaces and canvas states. Unlike saveLayer, however, save never creates a new surface.

Let’s wrap this pattern into a ClipRRect drawing command, which like SaveLayer takes a list of subcommands and a should_clip parameter indicating whether the clip is necessary:If you’re doing two clips at once, or a clip and a transform, or some other more complex setup that would benefit from only saving once but doing multiple things inside it, this pattern of always saving canvas parameters might be wasteful, but since it doesn’t create a surface it’s still a big optimization here.

Now, in paint_visual_effects, we can use ClipRRect instead of destination-in blending with DrawRRect (and we can fold the opacity into the skia.Paint passed to the outer SaveLayer, since that is defined to be applied before blending):

Of course, clipRRect only applies for rounded rectangles, while masking is a general technique that can be used to implement all sorts of clips and masks (like CSS’s clip-path and mask), so a real browser will typically have both code paths.

So now, each element uses at most one surface, and even then only if it has opacity or a non-default blend mode. Everything else should look visually the same, but will be faster and use less memory.

Browser compositing

Optimizing away surfaces is great when they’re not needed, but sometimes having more surfaces allows faster scrolling and animations. (In this section we’ll optimize scrolling; animations will have to wait for Chapter 13.)

So far, any time anything changed in the browser chrome or the web page itself, we had to clear the canvas and re-raster everything on it from scratch. This is inefficient—ideally, things should be re-rastered only if they actually change. When the context is complex or the screen is large, rastering too often produces a visible slowdown, and laptop and mobile batteries are drained unnecessarily. Real browsers optimize these situations by using a technique I’ll call browser compositing. The idea is to create a tree of explicitly cached surfaces for different pieces of content. Whenever something changes, we’ll re-raster only the surface where that content appears. Then these surfaces are blended (or “composited”) together to form the final image that the user sees.

Let’s implement this, with a surface for browser chrome and a surface for the current Tab’s contents. This way, we’ll only need to re-raster the Tab surface if page contents change, but not when (say) the user types into the address bar. This technique also allows us to scroll the Tab without any raster at all—we can just translate the page contents surface when drawing it.

To start with, we’ll need two new surfaces on Browser, chrome_surface and tab_surface:We could even use a different surface for each Tab, but real browsers don’t do this, since each surface uses up a lot of memory, and typically users don’t notice the small raster delay when switching tabs.

I’m not explicitly creating tab_surface right away, because we need to lay out the page contents to know how tall the surface needs to be.

Since we didn’t create the tab_surface on startup, we need to create it at the top of raster_tab:For a very big web page, the tab_surface can be much larger than the size of the SDL window, and therefore take up a very large amount of memory. We’ll ignore that, but a real browser would only paint and raster surface content up to a certain distance from the visible region, and re-paint/raster as the user scrolls.

The way we compute the page bounds here, based on the layout tree’s height, would be incorrect if page elements could stick out below (or to the right) of their parents—but our browser doesn’t support any features like that. Note that we need to recreate the tab surface if the page’s height changes.

Next, we need new code in draw to copy from the chrome and tab surfaces to the root surface. Moreover, we need to translate the tab_surface down by CHROME_PX and up by scroll, and clips it to only the area of the window that doesn’t overlap the browser chrome:

Finally, everywhere in Browser that we call draw, we now need to call either raster_tab or raster_chrome first. For example, in handle_click, we do this:

Notice how we don’t redraw the chrome when only the tab changes, and vice versa. In handle_down, which scrolls the page, we don’t need to call raster_tab at all, since scrolling doesn’t change the page.

We also have some related changes in Tab. First, we no longer need to pass around the scroll offset to the execute methods, or account for CHROME_PX, because we always draw the whole tab to the tab surface:

Likewise, we can remove the scroll parameter from each drawing command’s execute method:

Our browser now uses composited scrolling, making scrolling faster and smoother. In fact, in terms of conceptual phases of execution, our browser is now very close to real browsers: real browsers paint display lists, break content up into different rastered surfaces, and finally draw the tree of surfaces to the screen. There’s more we can do for performance—ideally we’d avoid all duplicate or unnecessary operations—but let’s leave that for the next few chapters.

Summary

So there you have it: our browser can draw not only boring text and boxes but also:

Besides the new features, we’ve upgraded from Tkinter to SDL and Skia, which makes our browser faster and more responsive, and also sets a foundation for more work on browser performance to come.

Outline

The complete set of functions, classes, and methods in our browser should now look something like this:

WIDTH

HEIGHT

HSTEP

VSTEP

SCROLL_STEP

def print_tree(node, indent)

class HTMLParser:
    def __init__(body)

    def parse()

    def get_attributes(text)

    def add_text(text)

    SELF_CLOSING_TAGS

    def add_tag(tag)

    HEAD_TAGS

    def implicit_tags(tag)

    def finish()

BLOCK_ELEMENTS

class DrawRect:
    def __init__(x1, y1, x2, y2, color)

    def execute(canvas)

    def __repr__()

class DocumentLayout:
    def __init__(node)

    def layout()

    def paint(display_list)

    def __repr__()

class CSSParser:
    def __init__(s)

    def whitespace()

    def literal(literal)

    def word()

    def pair()

    def ignore_until(chars)

    def body()

    def selector()

    def parse()

class TagSelector:
    def __init__(tag)

    def matches(node)

    def __repr__()

class DescendantSelector:
    def __init__(ancestor, descendant)

    def matches(node)

    def __repr__()

INHERITED_PROPERTIES

def style(node, rules)

def cascade_priority(rule)

class DrawText:
    def __init__(x1, y1, text, font, color)

    def execute(canvas)

    def __repr__()

def tree_to_list(tree, list)

class DrawLine:
    def __init__(x1, y1, x2, y2, color, thickness)

    def execute(canvas)

    def __repr__()

class DrawOutline:
    def __init__(x1, y1, x2, y2, color, thickness)

    def execute(canvas)

    def __repr__()

class LineLayout:
    def __init__(node, parent, previous)

    def layout()

    def paint(display_list)

    def __repr__()

class TextLayout:
    def __init__(node, word, parent, previous)

    def layout()

    def paint(display_list)

CHROME_PX

class Text:
    def __init__(text, parent)

    def __repr__()

class Element:
    def __init__(tag, attributes, parent)

    def __repr__()

class BlockLayout:
    def __init__(node, parent, previous)

    def token(tok)

    def word(node, word)

    def flush()

    def recurse(node)

    def open_tag(tag)

    def close_tag(tag)

    def layout()

    def layout_mode()

    def paint(display_list)

    def get_font(node)

    def __repr__()

    def new_line()

    def input(node)

class InputLayout:
    def __init__(node, parent, previous)

    def layout()

    def paint(display_list)

    def __repr__()

INPUT_WIDTH_PX

class Browser:
    def __init__()

    def handle_down()

    def handle_click(e)

    def handle_key(char)

    def handle_enter()

    def load(url)

    def paint_chrome()

    def draw()

    def raster_tab()

    def raster_chrome()

    def handle_quit()

EVENT_DISPATCH_CODE

COOKIE_JAR

class URL:
    def __init__(url)

    def request(top_level_url, payload)

    def resolve(url)

    def origin()

class JSContext:
    def __init__(tab)

    def run(code)

    def dispatch_event(type, elt)

    def get_handle(elt)

    def querySelectorAll(selector_text)

    def getAttribute(handle, attr)

    def innerHTML_set(handle, s)

    def XMLHttpRequest_send(method, url, body)

class Tab:
    def __init__()

    def load(url, body)

    def draw(canvas)

    def scrolldown()

    def click(x, y)

    def go_back()

    def __repr__()

    def render()

    def submit_form(elt)

    def keypress(char)

    def allowed_request(url)

    def raster(canvas)

FONTS

def get_font(size, weight, style)

def parse_color(color)

def parse_blend_mode(blend_mode_str)

def linespace(font)

class SaveLayer:
    def __init__(sk_paint, children, should_save, should_paint_cmds)

    def execute(canvas)

class DrawRRect:
    def __init__(rect, radius, color)

    def execute(canvas)

class ClipRRect:
    def __init__(rect, radius, children, should_clip)

    def execute(canvas)

def paint_visual_effects(node, cmds, rect)

if __name__ == "__main__"

If you run it, it should look something like this page; due to the browser sandbox, you will need to open that page in a new tab.

Exercises

CSS transforms: Add support for the transform CSS property, specifically the translate and rotate transforms.There is a lot more complexity to 3D transforms having to do with the definition of 3D spaces, flatting, backfaces, and plane intersections. Skia has built-in support for these via canvas state.

Filters: The filter CSS property allows specifying various kinds of more complex effects, such as grayscale or blur. These are fun to implement, and a number of them have built-in support in Skia. Implement, for example, the blur filter. Think carefully about when filters occur, relative to other effects like transparency, clipping, and blending.

Hit testing: If you have an element with a border-radius, it’s possible to click outside the element but inside its containing rectangle, by clicking in the part of the corner that is “rounded off”. This shouldn’t result in clicking on the element, but in our browser it currently does. Modify the click method to take border radii into account.

Interest region: Our browser now draws the whole web page to a single surface, and then shows parts of that surface as the user scrolls. That means a very long web page (like this one!) can create a large surface, thereby using a lot of memory. Modify the browser so that the height of that surface is limited, say to 4 * HEIGHT pixels. The (limited) region of the page drawn to this surface is called the interest region; you’ll need to track what part of the interest region is being shown on the screen, and re-raster the interest region when the user attempts to scroll outside of it.

One way to do this is to filter out all display list items that don’t intersect the interest rect. Another, easier way is to take advantage of Skia’s internal optimizations: if you call save and clipRect on a Skia canvas and then some draw operations, Skia will automatically avoid display item raster work outside of the clipping rectangle before the next restore.

Z-index: Right now, elements later in the HTML document are drawn “on top” of earlier ones. The z-index CSS property changes that order: an element with the larger z-index draws on top (with ties broken by the current order, and with the default z-index being 0). For z-index to have any effect, the element’s position property must be set to something other than static (the default). Add support for z-index. One thing you’ll run into is that with our browser’s minimal layout features, you might not be able to create any overlapping elements to test this feature! However, lots of exercises throughout the book allow you to create overlapping elements, including transform and width/height. For an extra challenge, add support for nested elements with z-index properties.

Overflow scrolling: An element with the overflow property set to scroll and a fixed pixel height is scrollable. (You’ll want to implement the width/height exercise from Chapter 6 so that height is supported.) Implement some version of overflow: scroll. I recommend the following user interaction: the user clicks within a scrollable element to focus it, and then can press the arrow keys to scroll up and down. You’ll need to keep track of the layout overflow. For an extra challenge, make sure you support scrollable elements nested within other scrollable elements.