dpy5 provides a Processing-inspired API (e.g., push(), pop(), fill(), stroke(), line(), curve()) for building differentiable 2D vector scenes. Under the hood it uses pydiffvg and PyTorch, so all parameters are tensors, and gradients can flow through the rendering process. It provides an "immediate mode" API on top of DiffVG, making it easier to experiment and build geometry through the composition of differentiable operations.
Internally, dpy5 caches primitives and concatenates transformations as drawing and styling functions are called. It then builds a scene from scratch when a render function is called. In practice, this is slightly slower than a retained-mode approach, where primitives are explicitly stored as object instances (e.g. a Path, a Circle). However, the speed hit is negligeable with respect to rendering time or the computation of image-space losses and the immediate-mode approach results in much simpler code and an easier way to build complex geometric compositions and transformation hierarchies.
dpy5 can be used standalone in Python scripts or Jupyter notebooks. The API is almost identical to Py5canvas, a Python port of Processing/P5js, so the two can be used together to create interactive sketches that take advantage of differentiable rasterization. You can convert a dpy5.DiffCanvas to a py5canvas.Canvas using the DiffCanvas.to_canvas() function, which then enables cleaner rendering using Cairo, easy saving to SVG or PDF, as well as copying outputs to the clipboard.
Installation should be easy within a conda environment, but installing [DiffVG]((https://github.com/BachiLi/diffvg) can be tricky depending on the platform.
Prerequisites:
You can install through PyPi using
pip install dpy5
To install the most recent version locally, clone this repository and then running
pip install -e .
from the repo directory.
import torch
from dpy5 import DiffCanvas
# Create a canvas (width, height)
c = DiffCanvas(256, 256)
c.background(1.0) # white background
c.fill(1.0, 0.0, 0.0, 1.0) # red
c.stroke(0.0) # black stroke
c.stroke_weight(2.0)
c.polyline([[50, 50], [200, 50], [200, 200], [50, 200]], close=True)
# Render the scene (differentiable, img contains the resulting tensor)
img = c.render() # Returnsaa a tensor with the rendered image
c.get_image()Calling render rasterizes the scene while allowing gradient propagation to any parameter used in the drawing procedures. By default, this operation also clears the DiffVG primitives that have been constructed during the previous drawing calls, so that subsequent sequences of commands dynamically rebuild the scene.
A typical optimization loop, involves re-drawing the scene at each step and using the otuput of render to compute some loss function with respect to the rendered image.
All drawing functions accept python sequences (e.g. lists, tuples, numpy arrays or pytorch tensors). Passing arguments as tensors with gradients enabled (requires_grad=True) will enable gradient propagation to the correponding variable. While you can do so explictly (e.g. c.polyline(some_tensor_with_grad)), the API allows you to do so more concisely by using the c.var(value, group_name) syntax.
The var method returns a PyTorch tensor with requires_grad=True and providing the second group_name argument will cache the tensor so it can be retrieved later together with all the tensors with the same group using c.get_vars(group_name).
The values passed into c.var will be used to initialize the tensor, but subsequent calls to the drawing sequence will use the cached tensor instead of re-creating new ones, as long as the same variable creation order is maintained.
Note: while this caching method saves typing, it expects the drawing order and tensor sizes to remain unchanged for each step of the optimization.
The following is an example that adapts a series of curves to minimize the L1 error with a target image and displays the results:
import os
import torch
import matplotlib.pyplot as plt
from dpy5 import DiffCanvas
from PIL import Image
from tqdm import tqdm
import numpy as np
target_img = Image.open('./spock256.jpg')
w, h = target_img.size
c = DiffCanvas(w, h)
# This function will be called repeatedly during optimization
# and the `c.var` variables will set on the first call to draw and then
# change during optimization
def draw(c):
c.background(1.0)
c.stroke(0); c.no_fill()
c.stroke_weight(2.0)
n_rows = 25
n_pts = 30
h = (c.height / n_rows)*0.1
for row_y in np.linspace(0, c.height, n_rows+2)[1:-1]:
x = np.linspace(0, c.width, n_pts)
y = row_y + c.var(np.random.uniform(-h, h, n_pts), 'offset')
c.curve(x, y)
c.render(prefiltering=True)
return c.img
# Initial image and target
draw(c)
initial_img = c.get_image()
target = c.to(np.array(target_img.convert('L'))/255)
# Optimization loop
optimizers = [
torch.optim.Adam(c.get_vars('offset'), lr=1.0),
]
for step in tqdm(range(250), 'Opt progress:'):
for opt in optimizers:
opt.zero_grad()
img = draw(c)
loss = (c.img.mean(dim=-1) - target).abs().mean() # L1
loss.backward()
for opt in optimizers:
opt.step()
# Display
plt.figure(figsize=(9,4))
plt.subplot(131)
plt.title('init'); plt.axis('off')
plt.imshow(initial_img)
plt.subplot(132)
plt.title('target'); plt.axis('off')
plt.imshow(target_img)
plt.subplot(133)
plt.title('optimizied'); plt.axis('off')
plt.imshow(c.get_image())
plt.tight_layout()
plt.show()
canvas = DiffCanvas(width, height, device=None)width,height: image size in pixels.device: PyTorch device (defaults to CUDA if available).
canvas.render(prefiltering=False, num_samples=2, seed=0, sdf=False)prefiltering: ifTrue, uses an anti‑aliasing prefilter. Produces crisper lines, but does not support variable width strokes and produces artefacts in some cases.num_samples: multisampling level.sdf: ifTrue, outputs a signed distance field.
After rendering, the result is stored in canvas.img. Retrieve it as a PIL image with canvas.get_image() or as a NumPy array with canvas.get_array().
| Method | Description |
|---|---|
fill(*args) |
Set fill color. Accepts 1-4 numbers/tensors. |
stroke(*args) |
Set stroke color. |
stroke_weight(w) |
Set line width. |
push() / pop() |
Save/restore transformation and style. |
push_matrix() / pop_matrix() |
Save/restore only the transformation matrix. |
push_style() / pop_style() |
Save/restore only style attributes. |
translate(x, y) |
Apply translation. |
rotate(angle) |
Apply rotation (in radians by default). |
scale(sx, sy) |
Apply scaling. |
identity() / reset_matrix() |
Reset the current transformation to identity. |
angle_mode(mode) |
Set angle mode: 'radians' or 'degrees'. |
rect_mode(mode) |
Set rectangle drawing mode: 'corner', 'corners', 'center', 'radius'. |
ellipse_mode(mode) |
Set ellipse drawing mode: 'corner', 'center', 'radius', 'corners'. |
fill_rule(rule) |
Set fill rule: 'evenodd', 'nonzero', 'winding'. |
curve_tightness(val) |
Set tension for cardinal splines (0‑1, default 0.5). |
| Method | Description |
|---|---|
line(x0, y0, x1, y1) or line([x0,y0], [x1,y1]) |
Draw a straight line. |
polyline(points, close=False) |
Draw an open or closed polyline. |
multibezier(points, close=False) |
Draw a sequence of cubic Bézier segments. |
curve(points, close=False) |
Draw a smooth cardinal spline through the given points. |
shape(obj, close=False) |
Draw a Shape object or a list of polylines. |
rect(x, y, w, h, radius=None) or rectangle(...) |
Draw a rectangle. Optional radius for rounded corners. |
square(x, y, size) |
Draw a square. |
ellipse(x, y, w, h) |
Draw an ellipse. |
circle(x, y, r) |
Draw a circle. |
triangle(a, b, c) |
Draw a triangle from three points. |
quad(a, b, c, d) |
Draw a quadrilateral from four points. |
| background(*args) | Clear the canvas with a color and begin a new scene. |
| no_fill() | Disable fill for subsequent shapes. |
| no_stroke() | Disable stroke for subsequent shapes. |
| `var(value, group_name, grad=True)= | Create/cache an optimization variable tensor. |
| =get_vars(group_name)= / =vars(group_name)= | Retrieve cached variables for a group. |
Build shapes piece by piece, similar to Processing’s beginShape() / endShape():
canvas.begin_shape()
canvas.begin_contour()
canvas.vertex(0, 0)
canvas.bezier_vertex(100, 50, 200, 50, 300, 0)
canvas.end_contour()
canvas.begin_contour()
canvas.curve_vertex(0, 200)
canvas.curve_vertex(100, 150)
canvas.curve_vertex(200, 150)
canvas.curve_vertex(300, 200)
canvas.end_contour(close=True)
canvas.end_shape()The Shape class can also be used standalone:
s = Shape(tension=0.5)
s.begin_shape()
s.vertex(...)
s.end_shape()
canvas.shape(s)Calling canvas.shape(s) multiple times will instance the same geometry with the current transformation, reusing the underlying pydiffvg paths.
CanvasOptimizer wraps a DiffCanvas and manages the optimization loop.
Subclass it and override draw, setup, loss
class MyOpt(CanvasOptimizer):
def draw(self, c):
c.background(1.0)
# ... build scene ...
return c.render()
def setup(self, c):
self.optimizers = [torch.optim.Adam(c.get_vars('pts'), lr=1.0)]
def loss(self, img):
return (img.mean(-1) - target).abs().mean()
opt = MyOpt(w, h, num_opt_steps=300)
opt.run()
for _ in range(opt.num_opt_steps):
opt.step()