Right now our browser can only draw colored rectangles and text—pretty boring! Real browsers support all kinds of visual effects that change how pixels and colors blend together. Let’s implement these effects using the Skia graphics library, and also see a bit of how Skia is implemented under the hood. That’ll also allow us to use surfaces for browser compositing to accelerate scrolling.
Before we get any further, we’ll need to upgrade our graphics system. While Tkinter is great for basic shapes and handling input, it lacks built-in support for many visual effects.That’s because Tk, the graphics library that Tkinter uses, dates from the early 90s, before high-performance graphics cards and GPUs became widespread. Implementing all details of the web’s many visual effects is fun, but it’s outside the scope of this book, so we need a new graphics library. Let’s use Skia, the library that Chromium uses. Unlike Tkinter, Skia doesn’t handle inputs or create graphical windows, so we’ll pair it with the SDL GUI library.
Start by installing Skia and SDL:
pip3 install skia-python pysdl2 pysdl2-dll
As elsewhere in this book, you may need to use
python3 -m pip instead of
pip3 as your installer, or use your IDE’s package
installer. If you’re on Linux, you’ll need to install additional
dependencies, like OpenGL and fontconfig. Also, you may not be able to
pysdl2-dll; if so, you’ll need to find SDL in your
system package manager instead. Consult the
web pages for more details.
Once installed, remove the
tkinter imports from browser
and replace them with these:
import ctypes import sdl2 import skia
If any of these imports fail, you probably need to check that Skia
and SDL were installed correctly. Note that the
module comes standard in Python; it is used to convert between Python
and C types.
Tkinter. Combined with WebGL,
Alternatively, one can compile Skia to
to do the same.
The main loop of the browser first needs some boilerplate to get SDL started:
if __name__ == "__main__": import sys sdl2.SDL_Init(sdl2.SDL_INIT_EVENTS)= Browser() browser 1])) browser.load(URL(sys.argv[# ...
Next, we need to create an SDL window, instead of a Tkinter window, inside the Browser, and set up Skia to draw to it. Here’s the SDL incantation to create a window:
class Browser: def __init__(self): self.sdl_window = sdl2.SDL_CreateWindow(b"Browser", sdl2.SDL_WINDOWPOS_CENTERED, sdl2.SDL_WINDOWPOS_CENTERED, WIDTH, HEIGHT, sdl2.SDL_WINDOW_SHOWN)
To set up Skia to draw to this window, we also need to create a surface for it:In Skia and SDL, a surface is a representation of a graphics buffer into which you can draw pixels (bits representing colors). A surface may or may not be bound to the physical pixels on the screen via a window, and there can be many surfaces. A canvas is an API interface that allows you to draw into a surface with higher-level commands such as for rectangles or text. Our browser uses separate Skia and SDL surfaces for simplicity, but in a highly optimized browser, minimizing the number of surfaces is important for good performance.
class Browser: def __init__(self): self.root_surface = skia.Surface.MakeRaster( skia.ImageInfo.Make( WIDTH, HEIGHT,=skia.kRGBA_8888_ColorType, ct=skia.kUnpremul_AlphaType)) at
Typically, we’ll draw to the Skia surface, and then once we’re done with it we’ll copy it to the SDL surface to display on the screen. This will be a little hairy, because we are moving data between two low-level libraries, but really it’s just copying pixels from one place to another.
First, get the sequence of bytes representing the Skia surface:
class Browser: def draw(self): # ... # This makes an image interface to the Skia surface, but # doesn't actually copy anything yet. = self.root_surface.makeImageSnapshot() skia_image = skia_image.tobytes() skia_bytes
Next, we need to copy the data to an SDL surface. This requires
telling SDL what order the pixels are stored in (which we specified to
RGBA_8888 when constructing the surface) and on your
class Browser: def __init__(self): if sdl2.SDL_BYTEORDER == sdl2.SDL_BIG_ENDIAN: self.RED_MASK = 0xff000000 self.GREEN_MASK = 0x00ff0000 self.BLUE_MASK = 0x0000ff00 self.ALPHA_MASK = 0x000000ff else: self.RED_MASK = 0x000000ff self.GREEN_MASK = 0x0000ff00 self.BLUE_MASK = 0x00ff0000 self.ALPHA_MASK = 0xff000000
CreateRGBSurfaceFrom method then wraps the data in
an SDL surface (this SDL surface does not copy the bytes): Note that since Skia and SDL
are C++ libraries, they are not always consistent with Python’s garbage
collection system. So the link between the output of
sdl_window is not guaranteed to be
kept consistent when
skia_bytes is garbage collected.
Instead, the SDL surface will be pointing at a bogus piece of memory,
which will lead to memory corruption or a crash. The code here is
correct because all of these are local variables that are
garbage-collected together, but if not you need to be careful to keep
all of them alive at the same time.
class Browser: def draw(self): # ... = 32 # Bits per pixel depth = 4 * WIDTH # Bytes per row pitch = sdl2.SDL_CreateRGBSurfaceFrom( sdl_surface skia_bytes, WIDTH, HEIGHT, depth, pitch,self.RED_MASK, self.GREEN_MASK, self.BLUE_MASK, self.ALPHA_MASK)
Finally, we draw all this pixel data on the window itself by blitting
(copying) it from
class Browser: def draw(self): # ... = sdl2.SDL_Rect(0, 0, WIDTH, HEIGHT) rect = sdl2.SDL_GetWindowSurface(self.sdl_window) window_surface # SDL_BlitSurface is what actually does the copy. sdl2.SDL_BlitSurface(sdl_surface, rect, window_surface, rect)self.sdl_window) sdl2.SDL_UpdateWindowSurface(
Next, SDL doesn’t have a
method; we have to implement it ourselves:
if __name__ == "__main__": # ... = sdl2.SDL_Event() event while True: while sdl2.SDL_PollEvent(ctypes.byref(event)) != 0: if event.type == sdl2.SDL_QUIT: browser.handle_quit() sdl2.SDL_Quit() sys.exit()# ...
The details of
too important here, but note that
SDL_QUIT is an event,
sent when the user closes the last open window. The
handle_quit method it calls just cleans up the window
class Browser: def handle_quit(self): self.sdl_window) sdl2.SDL_DestroyWindow(
We’ll also need to handle all of the other events in this loop—clicks, typing, and so on. The SDL syntax looks like this:
if __name__ == "__main__": while True: while sdl2.SDL_PollEvent(ctypes.byref(event)) != 0: # ... elif event.type == sdl2.SDL_MOUSEBUTTONUP: browser.handle_click(event.button)elif event.type == sdl2.SDL_KEYDOWN: if event.key.keysym.sym == sdl2.SDLK_RETURN: browser.handle_enter()elif event.key.keysym.sym == sdl2.SDLK_DOWN: browser.handle_down()elif event.type == sdl2.SDL_TEXTINPUT: 'utf8')) browser.handle_key(event.text.text.decode(
I’ve changed the signatures of the various event handler methods;
you’ll need to make analogous changes in
Browser where they
are defined. This loop replaces all of the
bind calls in
Browser constructor, which you can now remove.
SDL is most popular for making games. Their site lists a selection of books about game programming in SDL.
Now our browser is creating an SDL window and can draw to it via Skia. But most of the browser codebase is still using Tkinter drawing commands, which we now need to replace. Skia is a bit more verbose than Tkinter, so let’s abstract over some details with helper functions.Consult the Skia and skia-python documentation for more on the Skia API. First, a helper function to convert colors to Skia colors:
def parse_color(color): if color == "white": return skia.ColorWHITE elif color == "lightblue": return skia.ColorSetARGB(0xFF, 0xAD, 0xD8, 0xE6) # ... else: return skia.ColorBLACK
You can add more “elif” blocks to support any other color names you use; modern browsers support quite a lot.
To draw a line, you use Skia’s
class DrawLine: def execute(self, canvas, scroll): = skia.Path().moveTo(self.x1 - scroll, self.y1) \ path self.x2 - scroll, self.y2) .lineTo(= skia.Paint(Color=parse_color(self.color)) paint paint.setStyle(skia.Paint.kStroke_Style)self.thickness) paint.setStrokeWidth( canvas.drawPath(path, paint)
To draw text, you use
class DrawText: def execute(self, canvas, scroll): = skia.Paint( paint =True, Color=parse_color(self.color)) AntiAlias= self.top - scroll - self.font.getMetrics().fAscent baseline self.text, float(self.left), baseline, canvas.drawString(self.font, paint)
Finally, for drawing rectangles you use
Filling in the rectangle is the default:
class DrawRect: def execute(self, canvas, scroll): = skia.Paint() paint self.color)) paint.setColor(parse_color(self.rect.makeOffset(0, -scroll), paint) canvas.drawRect(
rect field is a Skia
which you can construct using
MakeLTRB (for “make
MakeXYWH (for “make
class DrawRect: def __init__(self, x1, y1, x2, y2, color): self.rect = skia.Rect.MakeLTRB(x1, y1, x2, y2) # ...
To draw just the outline, set the
Style parameter of the
Stroke_Style. Here “stroke” is a
standard term referring to drawing along the border of some shape; the
opposite is “fill”, meaning filling in the interior of the shape:
class DrawOutline: def execute(self, canvas): = skia.Paint() paint paint.setStyle(skia.Paint.kStroke_Style)self.thickness) paint.setStrokeWidth(self.color)) paint.setColor(parse_color(self.rect.makeOffset(0, -scroll), paint) canvas.drawRect(
If you look at the details of these helper methods, you’ll see that
they all use a Skia
Paint object to describe a shape’s
borders and colors. We’ll be seeing a lot more features of
Paint in this chapter.
While we’re here, let’s also add a
rect field to the
other drawing commands, replacing its
need some font changes in the browser UI, because Skia draws fonts a bit
differently from Tkinter. I had to adjust the y position of the
plus sign and less than signs to keep them centered in their boxes. Feel
free to adjust to make everything look good on your
class DrawText: def __init__(self, x1, y1, text, font, color): # ... self.rect = \ self.right, self.bottom) skia.Rect.MakeLTRB(x1, y1, class DrawLine: def __init__(self, x1, y1, x2, y2, color, thickness): # ... self.rect = skia.Rect.MakeLTRB(x1, y1, x2, y2)
Since we’re replacing Tkinter with Skia, we are also replacing
tkinter.font. In Skia, a font object has two pieces: a
Typeface, which is a type family with a certain weight,
style, and width; and a
Font, which is a
Typeface at a particular size. It’s the
Typeface that contains data and caches, so that’s what we
need to cache:
def get_font(size, weight, style): = (weight, style) key if key not in FONTS: if weight == "bold": = skia.FontStyle.kBold_Weight skia_weight else: = skia.FontStyle.kNormal_Weight skia_weight if style == "italic": = skia.FontStyle.kItalic_Slant skia_style else: = skia.FontStyle.kUpright_Slant skia_style = skia.FontStyle.kNormal_Width skia_width = \ style_info skia.FontStyle(skia_weight, skia_width, skia_style)= skia.Typeface('Arial', style_info) font = font FONTS[key] return skia.Font(FONTS[key], size)
Our browser also needs font metrics and measurements. In Skia, these
are provided by the
methods. Let’s start with
measureText replacing all calls
measure. For example, in the
Tab, we must do:
class InputLayout: def paint(self, display_list): if self.node.is_focused: = self.x + self.font.measureText(text) cx # ...
There are also
measure calls in
draw method on
Browser, in the
text method in
BlockLayout, and in the
layout method in
TextLayout. Update all of
them to use
Also, in the
layout method of
DrawText we make calls to the
method on fonts. In Skia, this method is called
and to get the ascent and descent we use
Note the negative sign when accessing the ascent. In Skia, ascent and
descent are positive if they go downward and negative if they go upward,
so ascents will normally be negative, the opposite of Tkinter. There’s
no analog for the
linespace field that Tkinter provides,
but you can use descent minus ascent instead:
def linespace(font): = font.getMetrics() metrics return metrics.fDescent - metrics.fAscent
You should now be able to run the browser again. It should look and behave just as it did in previous chapters, and it’ll probably feel faster, because Skia and SDL are faster than Tkinter. This is one advantage of Skia: since it is also used by the Chromium browser, we know it has fast, built-in support for all of the shapes we might need. And if the transition felt easy—well, that’s one of the benefits to abstracting over the drawing backend using a display list!
Font rasterization is surprisingly deep, with techniques such as subpixel rendering and hinting used to make fonts look better on lower-resolution screens. These techniques are much less necessary on high-pixel-density screens, though. It’s likely that eventually, all screens will be high-density enough to retire these techniques.
Let’s reward ourselves for the big refactor with a simple feature
that Skia enables: rounded corners of a rectangle via the
border-radius CSS property, like this:
<div style="border-radius: 10px; background: lightblue"> This is some example text. </div>
Which looks like this:If you’re very observant, you may notice that the text here
protrudes past the background by just a handful of pixels. This is the
correct default behavior, and can be modified by the
overflow CSS property, which we’ll see later this
border-radius requires drawing a rounded
rectangle, so let’s add a new
class DrawRRect: def __init__(self, rect, radius, color): self.rect = rect self.rrect = skia.RRect.MakeRectXY(rect, radius, radius) self.color = color def execute(self, scroll, canvas): = parse_color(self.color) sk_color self.rrect, canvas.drawRRect(=skia.Paint(Color=sk_color)) paint
Note that Skia supports
RRects, or rounded rectangles,
natively, so we can just draw one right to a canvas. Now we can draw
these rounded rectangles for the background:
class BlockLayout: def paint(self, display_list): if not is_atomic: if bgcolor != "transparent": = float( radius self.node.style.get("border-radius", "0px")[:-2]) display_list.append(DrawRRect(rect, radius, bgcolor))
Similar changes should be made to
Implementing high-quality raster libraries is very interesting in its own right—check out Real-Time Rendering for more.There is also Computer Graphics: Principles and Practice, which incidentally I remember buying—this is Chris speaking—back in the days of my youth (1992 or so). At the time I didn’t get much further than rastering lines and polygons (in assembly language!). These days you can do the same and more with Skia and a few lines of Python. These days, it’s especially important to leverage GPUs when they’re available, and browsers often push the envelope. Browser teams typically include or work closely with raster library experts: Skia for Chromium and Core Graphics for WebKit, for example. Both of these libraries are used outside of the browser, too: Core Graphics in iOS and macOS, and Skia in Android.
Skia, like the Tkinter canvas we’ve been using until now, is a rasterization library: it converts shapes like rectangles and text into pixels. Before we move on to Skia’s advanced features, let’s talk about how rasterization works at a deeper level. This will help to understand how exactly those features work.
You probably already know that computer screens are a 2D array of pixels. Each pixel contains red, green and blue lights,Actually, some screens contain pixels besides red, green, and blue, including white, cyan, or yellow. Moreover, different screens can use slightly different reds, greens, or blues; professional color designers typically have to calibrate their screen to display colors accurately. For the rest of us, the software still communicates with the display in terms of standard red, green, and blue colors, and the display hardware converts to whatever pixels it uses. or color channels, that can shine with an intensity between 0 (off) and 1 (fully on). By mixing red, green, and blue, which is formally known as the sRGB color space, any color in that space’s gamut can be made.The sRGB color space dates back to CRT displays. New technologies like LCD, LED, and OLED can display more colors, so CSS now includes syntax for expressing these new colors. All color spaces have a limited gamut of expressible colors. In a rasterization library, a 2D array of pixels like this is called a surface.Sometimes they are called bitmaps or textures as well, but these words connote specific CPU or GPU technologies for implementing surfaces. Since modern devices have lots of pixels, surfaces require a lot of memory, and we’ll typically want to create as few as possible.
The job of a rasterization library is to determine the red, green, and blue intensity of each pixel on the screen, based on the shapes—lines, rectangles, text—that the application wants to display. The interface for drawing shapes onto a surface is called a canvas; both Tkinter and Skia had canvas APIs. In Skia, each surface has an associated canvas that draws to that surface.
Screens use red, green, and blue color channels to match the three types of cone cells in a human eye. We take it for granted, but color standards like CIELAB derive from attempts to reverse-engineer human vision. These cone cells vary between people: some have more or fewer (typically an inherited condition carried on the X chromosome). Moreover, different people have different ratios of cone types and those cone types use different protein structures that vary in the exact frequency of green, red, and blue that they respond to. The study of color thus combines software, hardware, chemistry, biology, and psychology.
Drawing shapes quickly is already a challenge, but with multiple shapes there’s an additional question: what color should the pixel be when two shapes overlap? So far, our browser has only handled opaque shapes,It also hasn’t considered subpixel geometry or anti-aliasing, which also rely on color mixing. and the answer has been simple: take the color of the top shape. But now we need more nuance.
Many objects in nature are partially transparent: frosted glass, clouds, or colored paper, for example. Looking through one, you see multiple colors blended together. That’s also why computer screens work: the red, green, and blue lights blend together and appear to our eyes as another color. Designers use this effectMostly. Some more advanced blending modes on the web are difficult, or perhaps impossible, in real-world physics. in overlays, shadows, and tooltips, so our browser needs to support color mixing.
Color mixing means we need to think carefully about the order of operations. For example, consider black text on an orange background, placed semi-transparently over a white background. The text is gray while the background is yellow-orange. That’s due to blending: the text and the background are both partially transparent and let through some of the underlying white:
But importantly, the text isn’t orange-gray: even though the text is partially transparent, none of the orange shines through. That’s because the order matters: the text is first blended with the background; since the text is opaque, its blended pixels are black and overwrite the orange background. Only then is this black-and-orange image blended with the white background. Doing the operations in a different order would lead to dark-orange or black text.
To handle this properly, browsers apply blending not to individual shapes but to a tree of stacking contexts. Conceptually, each stacking context is drawn onto its own surface, and then blended into its parent stacking context. Rastering a web page requires a bottom-up traversal of the tree of stacking contexts: to raster a stacking context you first need to raster its contents, including its child stacking contexts, and then the whole contents need to be blended together into the parent.
To match this use pattern, in Skia, surfaces form a stack. You can push a new surface on the stack, raster things to it, and then pop it off by blending it with surface below. When traversing the tree of stacking contexts, you push a new surface onto the stack every time you recurse into a new stacking context, and pop-and-blend every time you return from a child stacking context to its parent.
In real browsers, stacking contexts are formed by HTML elements with certain styles, up to any descendants that themselves have such styles. The full definition is actually quite complicated, so in this chapter we’ll simplify by treating every layout object as a stacking context.
Mostly, elements form
a stacking context because of CSS properties that have something to
do with layering (like
z-index) or visual effects (like
mix-blend-mode). On the other hand, the
overflow property, which can make an element scrollable,
does not induce a stacking context, which I think was a mistake.While we’re at it, perhaps
scrollable elements should also be a containing
block for descendants. Otherwise, a scrollable element can have
non-scrolling children via properties like
situation is very complicated to handle in real browsers.
The reason is that inside a modern browser, scrolling is done on the GPU
by offsetting two surfaces. Without a stacking context the browser might
(depending on the web page structure) have to move around multiple
independent surfaces with complex paint orders, in lockstep, to achieve
scrolling. Fixed- and sticky-positioned elements also form stacking
contexts because of their interaction with scrolling.
Color mixing happens when multiple page elements overlap. The easiest
way that happens in our browser is child elements overlapping their
parents, like this:There
are many more ways elements can overlap in a real browser: the
negative margins, and so many more. But color mixing works the same way
<div style="background-color:orange"> Parent<div style="background-color:white;border-radius:5px">Child</div> Parent</div>
It looks like this:
Right now, the white rectangle completely obscures part of the orange
one; the two colors don’t really need to “mix”, and in fact it kind of
looks like two orange rectangles instead of an orange rectangle with a
white one on top. Now let’s make the white child element
semi-transparent, so the colors have to mix. In CSS, that requires
opacity property with a value somewhere between 0
(completely transparent) and 1 (totally opaque). With 50% opacity on the
white child element, it looks like this:
Notice that instead of being pure white, the child element now has a light-orange background color, resulting from orange and white mixing. Let’s implement this in our browser.
The way to mix colors in Skia is to first create two surfaces, and
then draw one into the other. The most convenient way to do that is with
saveLayer instead of
createSurface because Skia doesn’t actually promise to
create a new surface, if it can optimize that away. So what you’re
really doing with
saveLayer is telling Skia that there is a
new conceptual layer (“piece of paper”) on the stack. Skia’s terminology
distinguishes between a layer and a surface for this reason as well, but
for our purposes it makes sense to assume that each new layer comes with
a surface. and
# draw parent =skia.Paint(Alphaf=0.5)) canvas.saveLayer(paint# draw child canvas.restore()
We first draw the parent, then create a new surface with
saveLayer to draw the child into, and then when the
restore call is made the
saveLayer are used to mix the colors in the two
surfaces together. Here we’re using the
which describes the opacity as a floating-point number from 0 to 1.
restore are like a
pair of parentheses enclosing the child drawing operations. This means
our display list is no longer just a linear sequence of drawing
operations, but a tree. So in our display list, let’s represent
saveLayer with a
SaveLayer command that takes
a sequence of other drawing commands as an argument:
class SaveLayer: def __init__(self, sk_paint, children): self.sk_paint = sk_paint self.children = children self.rect = skia.Rect.MakeEmpty() for cmd in self.children: self.rect.join(cmd.rect) def execute(self, scroll, canvas): =self.sk_paint) canvas.saveLayer(paintfor cmd in self.children: cmd.execute(scroll, canvas) canvas.restore()
Now let’s look at how we can add this to our existing
paint method for
BlockLayouts. Right now, this
method draws a background and then recurses into its children, adding
each drawing command straight to the global display list. Let’s instead
add those drawing commands to a temporary list first:
class BlockLayout: def paint(self, display_list): =  cmds # ... if bgcolor != "transparent": # ... cmds.append(DrawRRect(rect, radius, bgcolor)) for child in self.children: child.paint(cmds)# ... display_list.extend(cmds)
Now, before we add our temporary command list to the overall
display list, we can use
SaveLayer to add transparency to
the whole element. I’m going to do this in a new
paint_visual_effects method, because we’ll want to make the
same changes to all of our other layout objects:
class BlockLayout: def paint(self, display_list): # ... = paint_visual_effects(self.node, cmds, rect) cmds display_list.extend(cmds)
paint_visual_effects, we’ll parse the opacity
value and construct the appropriate
def paint_visual_effects(node, cmds, rect): = float(node.style.get("opacity", "1.0")) opacity return [ =opacity), cmds) SaveLayer(skia.Paint(Alphaf ]
paint_visual_effects receives a list of
commands and returns another list of commands. It’s just that the output
list is always a single
SaveLayer command that wraps the
original content—which makes sense, because first we need to draw the
commands to a surface, and then apply transparency to it when
blending into the parent.
This blog post gives a really nice visual overview of many of the same concepts explored in this chapter, plus way more content about how a library such as Skia might implement features like raster sampling of vector graphics for lines and text, and interpolation of surfaces when their pixel arrays don’t match resolution or orientation. I highly recommend it.
Now let’s pause and explore how opacity actually works under the hood. Skia, SDL, and many other color libraries account for opacity with a fourth alpha value for each pixel.The difference between opacity and alpha can be confusing. Think of opacity as a visual effect applied to content, but alpha as a part of content. Think of alpha as implementation technique for representing opacity. An alpha of 0 means the pixel is fully transparent (meaning, no matter what the colors are, you can’t see them anyway), and an alpha of 1 means a fully opaque.
When a pixel with alpha overlaps another pixel, the final color is a
mix of their two colors. How exactly the colors are mixed is defined by
Paint objects. Of course, Skia is pretty complex,
but we can sketch these paint operations in Python as methods on an
class Pixel: def __init__(self, r, g, b, a): self.r = r self.g = g self.b = b self.a = a
When we apply a
Paint with an
parameter, the first thing Skia does is add the requested opacity to
class Pixel: def alphaf(self, opacity): self.a = self.a * opacity
I want to emphasize that this code is not a part of our browser—I’m simply using Python code to illustrate what Skia is doing internally.
Alphaf operation applies to pixels in one surface.
SaveLayer we will end up with two surfaces, with
all of their pixels aligned, and therefore we will need to combine, or
blend, corresponding pairs of pixels.
Here the terminology can get confusing: we imagine that the pixels “on top” are blending into the pixels “below”, so we call the top surface the source surface, with source pixels, and the bottom surface the destination surface, with destination pixels. When we combine them, there are lots of ways we could do it, but the default on the web is called “simple alpha compositing” or source-over compositing. In Python, the code to implement it looks like this:The formula for this code can be found here. Note that that page refers to premultiplied alpha colors, but Skia’s API does not use premultiplied representations, and the code below doesn’t either.
class Pixel: def source_over(self, source): self.a = 1 - (1 - source.a) * (1 - self.a) if self.a == 0: return self self.r = \ self.r * (1 - source.a) * self.a + \ (* source.a) / self.a source.r self.g = \ self.g * (1 - source.a) * self.a + \ (* source.a) / self.a source.g self.b = \ self.b * (1 - source.a) * self.a + \ (* source.a) / self.a source.b
Here the destination pixel
self is modified to blend in
the source pixel
source. The mathematical expressions for
the red, green, and blue color channels are identical, and basically
average the source and destination colors, weighted by alpha.For example, if the alpha of
the source pixel is 1, the result is just the source pixel color, and if
it is 0 the result is the backdrop pixel color. You might
imagine the overall operation of
SaveLayer with an
Alphaf parameter as something like this:In reality, reading individual
pixels into memory to manipulate them like this is slow. So libraries
such as Skia don’t make it convenient to do so. (Skia canvases do have
readPixels methods that are
sometimes used, but not for this.)
for (x, y) in destination.coordinates(): source[x, y].alphaf(opacity) destination[x, y].source_over(source[x, y])
Source-over compositing is one way to combine two pixel values. But it’s not the only method—you could write literally any computation that combines two pixel values if you wanted. Two computations that produce interesting effects are traditionally called “multiply” and “difference” and use simple mathematical operations. “Multiply” multiplies the color values:
class Pixel: def multiply(self, source): self.r = self.r * source.r self.g = self.g * source.g self.b = self.b * source.b
And “difference” computes their absolute differences:
class Pixel: def difference(self, source): self.r = abs(self.r - source.r) self.g = abs(self.g - source.g) self.b = abs(self.b - source.b)
CSS supports these and many other blending modesMany of these blending modes
are common to
other graphics editing programs like Photoshop and GIMP. Some, like “dodge” and
“burn”, go back to analog photography, where photographers would
expose some parts of the image more than others to manipulate their
brightness. via the
property, like this:
<div style="background-color:orange"> Parent<div style="background-color:blue;mix-blend-mode:difference"> Child</div> Parent</div>
This HTML will look like:
Here, when blue overlaps with orange, we see pink: blue has (red,
green, blue) color channels of
(0, 0, 1), and orange has
(1, .65, 0), so with “difference” blending the resulting
pixel will be
(1, 0.65, 1), which is pink. On a pixel
level, what’s happening is something like this:
for (x, y) in destination.coordinates(): source[x, y].alphaf(opacity) source[x, y].difference(destination[x, y]) destination[x, y].source_over(source[x, y])
This looks weird, but conceptually it blends the destination into the source (which ignores alpha) and then draws the source over the destination (with alpha considered). In some sense, blending thus happens twice.
Skia supports the multiply and difference blend modes natively:
def parse_blend_mode(blend_mode_str): if blend_mode_str == "multiply": return skia.BlendMode.kMultiply elif blend_mode_str == "difference": return skia.BlendMode.kDifference else: return skia.BlendMode.kSrcOver
This makes adding support for blend modes to our browser as simple as
BlendMode parameter to the
def paint_visual_effects(node, cmds, rect): # ... = parse_blend_mode(node.style.get("mix-blend-mode")) blend_mode return [ =blend_mode), [ SaveLayer(skia.Paint(BlendMode=opacity), cmds), SaveLayer(skia.Paint(Alphaf ]), ]
Note the order of operations here: we first apply
transparency, and then blend the result into the rest of the
page. If we switched the two
SaveLayer calls, so that we
first applied blending, there wouldn’t be anything to blend it into!
Alpha might seem intuitive, but it’s less obvious than you think:
see, for example, this history of
alpha written by its co-inventor (and co-founder of Pixar). And
there are several different implementation options. For example, many
graphics libraries, Skia included, multiply the color channels by the
opacity instead of allocating a whole color channel. This premultiplied
representation is generally more efficient; for example,
source_over above had to divide by
the end, because otherwise the result would be premultiplied. Using a
premultiplied representation throughout would save a division. Nor is it
obvious how alpha behaves when
The “multiply” and “difference” blend modes can seem kind of obscure, but blend modes are a flexible way to implement per-pixel operations. One common use case is clipping—intersecting a surface with a given shape. It’s called clipping because it’s like putting a second piece of paper (called a mask) over the first one, and then using scissors to cut along the mask’s edge.
There are all sorts of powerful methodsThe CSS
property lets specify a mask shape using a curve, while the
property lets you instead specify a image URL for the
mask. for clipping content on the web, but the most common
form involves the
overflow property. This property has lots
of possible values,For
overflow: scroll adds scroll bars and makes an
element scrollable, while
overflow: hidden is similar to
but subtly different from
overflow: clip. but
let’s focus here on
overflow: clip, which cuts off contents
of an element that are outside the element’s bounds.
overflow: clip is used with properties like
rotate which can make an element’s
children poke outside their parent. Our browser doesn’t support these,
but there is one edge case where
overflow: clip is
relevant: rounded corners. Consider this example:
<div style="border-radius:30px;background-color:lightblue;overflow:clip"> This test text exists here to ensure that the "div" element is large enough that the border radius is obvious.</div>
That HTML looks like this:
Observe that the letters near the corner are cut off to maintain a
sharp rounded edge. (Uhh… actually, at the time of this writing, Safari
does not support
overflow: clip, so if you’re using Safari
you won’t see this effect.The similar
overflow: hidden is supported by
all browsers. However, in this case,
overflow: hidden will
also increase the height of
div until the rounded corners
no longer clip out the text. This is because
overflow:hidden has different rules for sizing boxes,
having to do with the possibility of the child content being
hidden means “clipped, but might be scrolled by
been impossible to see the text, which is really bad if it’s intended
that there should be a way to scroll it on-screen.) That’s
clipping; without the
overflow: clip property these letters
would instead be fully drawn, like we saw earlier in this chapter.
Counterintuitively, we’ll implement clipping using blending modes. We’ll make a new surface (the mask), draw a rounded rectangle into it, and then blend it with the element contents. But we want to see the element contents, not the mask, so when we do this blending we will use destination-in compositing.
Destination-in compositing basically means keeping the pixels of the destination surface that intersect with the source surface. The source surface’s color is not used—just its alpha. In our case, the source surface is the rounded rectangle mask and the destination surface is the content we want to clip, so destination-in fits perfectly. In code, destination-in looks like this:
class Pixel: def destination_in(self, source): self.a = self.a * source.a if self.a == 0: return self self.r = (self.r * self.a * source.a) / self.a self.g = (self.g * self.a * source.a) / self.a self.b = (self.b * self.a * source.a) / self.a
paint_visual_effects, we need to create a new
layer, draw the mask image into it, and then blend it with the element
contents with destination-in blending:
def paint_visual_effects(node, cmds, rect): # ... = float(node.style.get("border-radius", "0px")[:-2]) border_radius if node.style.get("overflow", "visible") == "clip": = border_radius clip_radius else: = 0 clip_radius return [ =blend_mode), [ SaveLayer(skia.Paint(BlendMode=opacity), cmds), SaveLayer(skia.Paint(Alphaf=skia.kDstIn), [ SaveLayer(skia.Paint(BlendMode"white") DrawRRect(rect, clip_radius, ]), ]), ]
After drawing all of the element contents with
applying opacity), this code draws a rounded rectangle on another layer
to serve as the mask, and uses destination-in blending to clip the
element contents. Here I chose to draw the rounded rectangle in white,
but the color doesn’t matter as long as it’s opaque. On the other hand,
if there’s no clipping, I don’t round the corners of the mask, which
means nothing is clipped out.
Notice how similar this masking technique is to the physical analogy with scissors described earlier, with the two layers playing the role of two sheets of paper and destination-in compositing playing the role of the scissors. This implementation technique for clipping is called masking, and it is very general—you can use it with arbitrarily complex mask shapes, like text, bitmap images, or anything else you can imagine.
Rounded corners have an interesting
history in computing. Features that are simple today were very
complex to implement on early personal computers with limited memory
and no hardware floating-point arithmetic. Even when floating-point
hardware and eventually GPUs became standard, the
border-radius CSS property didn’t appear in browsers until
around 2010.The lack of
support didn’t stop web developers from putting rounded corners on their
border-radius was supported. There are a
number of clever ways to do it; this
video walks through several. More recently, the
introduction of animations, visual effects, multi-process compositing,
overlays have again rounded corners pretty complex. The
clipRRect fast path, for example, can fail to apply for
cases such as hardware video overlays and nested rounded corner
Our browser now works correctly, but uses way too many surfaces. For example, for a single, no-effects-needed div with some text content, there are currently 18 surfaces allocated in the display list. If there’s no blending going on, we should only need one!
Let’s review all the surfaces that our code can create for an element:
But not every element has opacity, blend modes, or clipping applied,
and we could skip creating those surfaces most of the time. To implement
this without making the code hard to read, let’s change
SaveLayer to take two additional optional parameters:
saveLayer is called and whether subcommands
are actually painted:
class SaveLayer: def __init__(self, sk_paint, children, =True, should_paint_cmds=True): should_saveself.should_save = should_save self.should_paint_cmds = should_paint_cmds # ... def execute(self, canvas): if self.should_save: =self.sk_paint) canvas.saveLayer(paintif self.should_paint_cmds: for cmd in self.children: cmd.execute(canvas)if self.should_save: canvas.restore()
Turn off those parameters if an effect isn’t applied:
def paint_visual_effects(node, cmds, rect): # ... = node.style.get("overflow", "visible") == "clip" needs_clip = blend_mode != skia.BlendMode.kSrcOver or \ needs_blend_isolation needs_clip= opacity != 1.0 needs_opacity return [ =blend_mode), [ SaveLayer(skia.Paint(BlendMode=opacity), cmds, SaveLayer(skia.Paint(Alphaf=needs_opacity), should_save=skia.kDstIn), [ SaveLayer(skia.Paint(BlendMode"white") DrawRRect(rect, clip_radius, =needs_clip, should_paint_cmds=needs_clip), ], should_save=needs_blend_isolation), ], should_save ]
Now simple web pages always use a single surface—a huge saving in memory. But we can save even more surfaces. For example, what if there is a blend mode and opacity at the same time: can we use the same surface? Indeed, yes you can! That’s also pretty simple:This works for opacity, but not for filters that “move pixels” such as blur. Such a filter needs to be applied before clipping, not when blending into the parent surface. Otherwise, the edge of the blur will not be sharp.
def paint_visual_effects(node, cmds, rect): # ... = node.style.get("overflow", "visible") == "clip" needs_clip = blend_mode != skia.BlendMode.kSrcOver or \ needs_blend_isolation needs_clip= opacity != 1.0 needs_opacity return [ =blend_mode, Alphaf=opacity), SaveLayer(skia.Paint(BlendMode+ [ cmds =skia.kDstIn), [ SaveLayer(skia.Paint(BlendMode"white") DrawRRect(rect, clip_radius, =needs_clip, should_paint_cmds=needs_clip), ], should_save=needs_blend_isolation or needs_opacity), ], should_save ]
There’s one more optimization to make: using Skia’s
clipRRect operation to get rid of the destination-in
blended surface. This operation takes in a rounded rectangle and changes
the canvas state so that all future commands skip drawing any
pixels outside that rounded rectangle.
There are multiple advantages to using
clipRRect over an
explicit destination-in surface. First, most of the time, it allows Skia
to avoid making a surface for the mask.Typically in a browser this
means code in GPU shaders. GPU programs are out of scope for this book,
but if you’re curious there are many online resources describing ways to
do this. It also allows Skia to skip draw operations that
don’t intersect the mask, or dynamically draw only the parts of
operations that intersect it. It’s basically the optimization we
implemented for scrolling in
Chapter 2.This kind
of code is complex for Skia to implement, so it only makes sense to do
it for common patterns, like rounded rectangles. This is why Skia only
supports optimized clips for a few common shapes.
clipRRect changes the canvas state, we’ll need to
restore it once we’re done with clipping. That uses the
restore methods—you call
save before calling
restore after finishing drawing the commands that should be
# Draw commands that should not be clipped. canvas.save() canvas.clipRRect(rounded_rect) # Draw commands that should be clipped. canvas.restore() # Draw commands that should not be clipped.
If you’ve noticed that
restore is used for both saving
state and pushing surfaces, that’s because Skia has a combined stack of
surfaces and canvas states. Unlike
save never creates a new surface.
Let’s wrap this pattern into a
command, which like
SaveLayer takes a list of subcommands
should_clip parameter indicating whether the clip is
doing two clips at once, or a clip and a transform, or some other more
complex setup that would benefit from only saving once but doing
multiple things inside it, this pattern of always saving canvas
parameters might be wasteful, but since it doesn’t create a surface it’s
still a big optimization here.
class ClipRRect: def __init__(self, rect, radius, children, should_clip=True): self.rect = rect self.rrect = skia.RRect.MakeRectXY(rect, radius, radius) self.children = children self.should_clip = should_clip def execute(self, canvas): if self.should_clip: canvas.save()self.rrect) canvas.clipRRect( for cmd in self.children: cmd.execute(canvas) if self.should_clip: canvas.restore()
paint_visual_effects, we can use
ClipRRect instead of destination-in blending with
DrawRRect (and we can fold the opacity into the
skia.Paint passed to the outer
since that is defined to be applied before blending):
def paint_visual_effects(node, cmds, rect): # ... return [ =blend_mode, Alphaf=opacity), [ SaveLayer(skia.Paint(BlendMode ClipRRect(rect, clip_radius, cmds,=needs_clip), should_clip=needs_blend_isolation), ], should_save ]
clipRRect only applies for rounded
rectangles, while masking is a general technique that can be used to
implement all sorts of clips and masks (like CSS’s
mask), so a real browser will
typically have both code paths.
So now, each element uses at most one surface, and even then only if it has opacity or a non-default blend mode. Everything else should look visually the same, but will be faster and use less memory.
Besides using fewer surfaces, real browsers also need to avoid surfaces getting too big. Real browsers use tiling for this, breaking up the surface into a grid of tiles which have their own raster surfaces and their own x and y offset to the page. Whenever content that intersects a tile changes its display list, the tile is re-rastered. Tiles that are not on or “near”For example, tiles that just scrolled off-screen. the screen are not rastered at all. This all happens on the GPU, since surfaces (Skia ones in particular) can be stored on the GPU.
Optimizing away surfaces is great when they’re not needed, but sometimes having more surfaces allows faster scrolling and animations. (In this section we’ll optimize scrolling; animations will have to wait for Chapter 13.)
So far, any time anything changed in the browser chrome or the web page itself, we had to clear the canvas and re-raster everything on it from scratch. This is inefficient—ideally, things should be re-rastered only if they actually change. When the context is complex or the screen is large, rastering too often produces a visible slowdown, and laptop and mobile batteries are drained unnecessarily. Real browsers optimize these situations by using a technique I’ll call browser compositing. The idea is to create a tree of explicitly cached surfaces for different pieces of content. Whenever something changes, we’ll re-raster only the surface where that content appears. Then these surfaces are blended (or “composited”) together to form the final image that the user sees.
Let’s implement this, with a surface for browser chrome and a surface
for the current
Tab’s contents. This way, we’ll only need
to re-raster the
Tab surface if page contents change, but
not when (say) the user types into the address bar. This technique also
allows us to scroll the
Tab without any raster at all—we
can just translate the page contents surface when drawing it.
To start with, we’ll need two new surfaces on
tab_surface:We could even use a different
surface for each
Tab, but real browsers don’t do this,
since each surface uses up a lot of memory, and typically users don’t
notice the small raster delay when switching tabs.
class Browser: def __init__(self): # ... self.chrome_surface = skia.Surface(WIDTH, CHROME_PX) self.tab_surface = None
I’m not explicitly creating
tab_surface right away,
because we need to lay out the page contents to know how tall the
surface needs to be.
We’ll also need to split the browser’s
draw method into
drawwill composite the chrome and tab surfaces and copy the result from Skia to SDL;
raster_tabwill draw the page to the
raster_chromewill draw the browser chrome to the
Let’s start by doing the split:
class Browser: def raster_tab(self): = self.tab_surface.getCanvas() canvas canvas.clear(skia.ColorWHITE)# ... def raster_chrome(self): = self.chrome_surface.getCanvas() canvas canvas.clear(skia.ColorWHITE)# ... def draw(self): = self.root_surface.getCanvas() canvas canvas.clear(skia.ColorWHITE)# ...
Since we didn’t create the
tab_surface on startup, we
need to create it at the top of
raster_tab:For a very big web page, the
tab_surface can be much larger than the size of the SDL
window, and therefore take up a very large amount of memory. We’ll
ignore that, but a real browser would only paint and raster surface
content up to a certain distance from the visible region, and
re-paint/raster as the user scrolls.
import math class Browser: def raster_tab(self): = self.tabs[self.active_tab] active_tab = math.ceil(active_tab.document.height) tab_height if not self.tab_surface or \ != self.tab_surface.height(): tab_height self.tab_surface = skia.Surface(WIDTH, tab_height) # ...
The way we compute the page bounds here, based on the layout tree’s height, would be incorrect if page elements could stick out below (or to the right) of their parents—but our browser doesn’t support any features like that. Note that we need to recreate the tab surface if the page’s height changes.
Next, we need new code in
draw to copy from the chrome
and tab surfaces to the root surface. Moreover, we need to translate the
tab_surface down by
CHROME_PX and up by
scroll, and clips it to only the area of the window that
doesn’t overlap the browser chrome:
class Browser: def draw(self): # ... = skia.Rect.MakeLTRB(0, CHROME_PX, WIDTH, HEIGHT) tab_rect = CHROME_PX - self.tabs[self.active_tab].scroll tab_offset canvas.save() canvas.clipRect(tab_rect)0, tab_offset) canvas.translate(self.tab_surface.draw(canvas, 0, 0) canvas.restore() = skia.Rect.MakeLTRB(0, 0, WIDTH, CHROME_PX) chrome_rect canvas.save() canvas.clipRect(chrome_rect)self.chrome_surface.draw(canvas, 0, 0) canvas.restore() # ...
Finally, everywhere in
Browser that we call
draw, we now need to call either
raster_chrome first. For example, in
handle_click, we do this:
class Browser: def handle_click(self, e): if e.y < CHROME_PX: # ... self.raster_chrome() else: # ... self.raster_tab() self.draw()
Notice how we don’t redraw the chrome when only the tab changes, and
vice versa. In
handle_down, which scrolls the page, we
don’t need to call
raster_tab at all, since scrolling
doesn’t change the page.
We also have some related changes in
Tab. First, we no
longer need to pass around the scroll offset to the
methods, or account for
CHROME_PX, because we always draw
the whole tab to the tab surface:
class Tab: def raster(self, canvas): for cmd in self.display_list: cmd.execute(canvas)
Likewise, we can remove the
scroll parameter from each
class DrawRect: def execute(self, canvas): = skia.Paint() paint self.color)) paint.setColor(parse_color(self.rect, paint) canvas.drawRect(
Our browser now uses composited scrolling, making scrolling faster and smoother. In fact, in terms of conceptual phases of execution, our browser is now very close to real browsers: real browsers paint display lists, break content up into different rastered surfaces, and finally draw the tree of surfaces to the screen. There’s more we can do for performance—ideally we’d avoid all duplicate or unnecessary operations—but let’s leave that for the next few chapters.
Real browsers allocate new surfaces for various different situations,
such as implementing accelerated overflow scrolling and animations of
certain CSS properties such as transform
and opacity that can be done without raster. They also allow scrolling
arbitrary HTML elements via
in CSS. Basic scrolling for DOM elements is very similar to what we’ve
just implemented. But implementing it in its full generality, and with
excellent performance, is extremely challenging. Scrolling is
probably the single most complicated feature in a browser rendering
engine. The corner cases and subtleties involved are almost endless.
So there you have it: our browser can draw not only boring text and boxes but also:
Besides the new features, we’ve upgraded from Tkinter to SDL and Skia, which makes our browser faster and more responsive, and also sets a foundation for more work on browser performance to come.
The complete set of functions, classes, and methods in our browser should now look something like this:
def print_tree(node, indent)
def __init__(x1, y1, x2, y2, color)
def __init__(ancestor, descendant)
def style(node, rules)
def __init__(x1, y1, text, font, color)
def tree_to_list(tree, list)
def __init__(x1, y1, x2, y2, color, thickness)
def __init__(x1, y1, x2, y2, color, thickness)
def __init__(node, parent, previous)
def __init__(node, word, parent, previous)
def __init__(text, parent)
def __init__(tag, attributes, parent)
def __init__(node, parent, previous)
def word(node, word)
def __init__(node, parent, previous)
def request(top_level_url, payload)
def dispatch_event(type, elt)
def getAttribute(handle, attr)
def innerHTML_set(handle, s)
def XMLHttpRequest_send(method, url, body)
def load(url, body)
def click(x, y)
def get_font(size, weight, style)
def __init__(sk_paint, children, should_save, should_paint_cmds)
def __init__(rect, radius, color)
def __init__(rect, radius, children, should_clip)
def paint_visual_effects(node, cmds, rect)
if __name__ == "__main__"
If you run it, it should look something like this page; due to the browser sandbox, you will need to open that page in a new tab.
CSS transforms: Add support for the transform
CSS property, specifically the
rotate transforms.There is a lot more complexity to 3D transforms having to
do with the definition of 3D spaces, flatting, backfaces, and plane
intersections. Skia has built-in support for these via
filter CSS property allows
specifying various kinds of more complex
effects, such as grayscale or blur. These are fun to implement, and
a number of them have built-in support in Skia. Implement, for example,
blur filter. Think carefully about when filters occur,
relative to other effects like transparency, clipping, and blending.
Hit testing: If you have an element with a
border-radius, it’s possible to click outside the element
but inside its containing rectangle, by clicking in the part of the
corner that is “rounded off”. This shouldn’t result in clicking on the
element, but in our browser it currently does. Modify the
click method to take border radii into account.
Interest region: Our browser now draws the whole web page to
a single surface, and then shows parts of that surface as the user
scrolls. That means a very long web page (like this one!) can create a
large surface, thereby using a lot of memory. Modify the browser so that
the height of that surface is limited, say to
4 * HEIGHT
pixels. The (limited) region of the page drawn to this surface is called
the interest region; you’ll need to track what part of the interest
region is being shown on the screen, and re-raster the interest region
when the user attempts to scroll outside of it.
One way to do this is to filter out all display list items that don’t
intersect the interest rect. Another, easier way is to take advantage of
Skia’s internal optimizations: if you call
clipRect on a Skia canvas and then some draw operations,
Skia will automatically avoid display item raster work outside of the
clipping rectangle before the next
Z-index: Right now, elements later in the HTML document are
drawn “on top” of earlier ones. The
z-index CSS property
changes that order: an element with the larger
draws on top (with ties broken by the current order, and with the
z-index being 0). For
z-index to have
any effect, the element’s
position property must be set to
something other than
static (the default). Add support for
z-index. One thing you’ll run into is that with our
browser’s minimal layout features, you might not be able to
create any overlapping elements to test this feature! However,
lots of exercises throughout the book allow you to create overlapping
height. For an extra challenge, add
support for nested
Overflow scrolling: An element with the
overflow property set to
scroll and a fixed
height is scrollable. (You’ll want to implement the
width/height exercise from Chapter 6
height is supported.) Implement some version of
overflow: scroll. I recommend the following user
interaction: the user clicks within a scrollable element to focus it,
and then can press the arrow keys to scroll up and down. You’ll need to
keep track of the layout
overflow. For an extra challenge, make sure you support
scrollable elements nested within other scrollable elements.
Did you find this chapter useful?