Use SDF grids and surfels

Trace rays against signed-distance grids and Gaussian surfels, composite surfel opacity, and mix meshes, SDFs, and surfels in one MixedScene.

Besides triangle meshes, RayD traces two kinds of implicit geometry: signed-distance grids (SDFs), sampled on a regular 3D grid, and surfels, flat Gaussian discs with an opacity. Both are differentiable, so you can optimize an SDF or a surfel cloud through ray queries. MixedScene combines them with meshes.

Signed-distance grids#

An SdfGrid is a box of nx * ny * nz distance samples. The values are vertex-centered and negative inside the surface, and they are distances in world units. The box is centered at position, rotated by the scalar-first quaternion rotation = (qw, qx, qy, qz), and has full side lengths scale.

The Dr.Jit grid takes a flat Float in x-major order: sample (i, j, k) is at index (i * ny + j) * nz + k.

import math
import rayd.drjit as rd
from drjit.cuda import Array3f, Array3i, Float
 
n = 24
lin = [-1.0 + 2.0 * k / (n - 1) for k in range(n)]
values = [math.sqrt(x * x + y * y + z * z) - 0.5 for x in lin for y in lin for z in lin]
sphere = rd.SdfGrid(Float(values), n, n, n,
                    Array3f(0.0, 0.0, 0.0),        # position
                    Float([1.0, 0.0, 0.0, 0.0]),   # rotation (qw, qx, qy, qz)
                    Array3f(2.0, 2.0, 2.0))        # scale: a 2 x 2 x 2 box
 
hit = sphere.intersect(rd.Ray(Array3f([0.0], [0.0], [-2.0]), Array3f([0.0], [0.0], [1.0])))
print(hit.t, hit.hit_mask, hit.steps)   # [1.50375] [True] [5]
print(hit.normal)                       # about [[0, 0, -1]]

The SDF is traced by sphere tracing, with trilinear interpolation between samples. SdfTraceOptions controls it: max_steps (default 64), relaxation (default 0.9, the fraction of the distance to step), and eps_hit (the hit threshold; the default derives it from the grid). Besides intersect, a grid answers visible(start, end) and trace_reflections(ray, max_bounces).

The hit distance and position are differentiable with respect to the SDF values, the placement, and the ray. Make values an AD array (drjit.cuda.ad.Float) or a tensor with requires_grad=True.

Surfels#

A surfel is a flat Gaussian disc with a center, two tangent axes tangent_u and tangent_v that set its orientation and extent, an opacity, and one scalar value. Build a SurfelCloud, wrap it in a SurfelScene, and call build(). RayD encloses each surfel in a proxy shape (an icosahedron by default, so triangle_count is 20 per surfel) for hardware ray tracing.

cloud = rd.SurfelCloud(
    Array3f([0.0, 0.0], [0.0, 0.0], [0.0, -0.5]),   # centers
    Array3f([1.0, 1.0], [0.0, 0.0], [0.0, 0.0]),    # tangent_u
    Array3f([0.0, 0.0], [1.0, 1.0], [0.0, 0.0]),    # tangent_v
    Float([0.6, 0.6]))                              # opacity
surfels = rd.SurfelScene(cloud)
surfels.build()
print(surfels.surfel_count, surfels.triangle_count)   # 2 40
 
ray = rd.Ray(Array3f([0.0], [0.0], [1.0]), Array3f([0.0], [0.0], [-1.0]))
comp = surfels.composite_alpha(ray)
print(comp.alpha, comp.transmittance)   # [0.84] [0.16]: 1 - 0.4 * 0.4

intersect returns the first surfel hit, with t, local_uv, gaussian_weight, alpha, and surfel_id. composite_alpha blends every surfel along the ray front to back and returns the accumulated alpha, transmittance, and depth. visible and shadow_test treat the cloud as an occluder. SurfelTraceOptions sets the opacity cutoff (alpha_min, default 1/255), the per-hit cap (alpha_cap, 0.99), the early stop (transmittance_min, 0.03), and the proxy shape (primitive_mode).

Render colors and features (Dr.Jit)#

For appearance fitting, split geometry and appearance: build the scene from a SurfelGeometry, attach a SurfelAppearance, and call render.

geo = rd.SurfelGeometry(Array3f([0.0, 0.0], [0.0, 0.0], [0.0, -0.5]),
                        Array3f([1.0, 1.0], [0.0, 0.0], [0.0, 0.0]),
                        Array3f([0.0, 0.0], [1.0, 1.0], [0.0, 0.0]))
scene = rd.SurfelScene(geo)
scene.update_appearance(rd.SurfelAppearance.rgb(Float([0.6, 0.6]),
                                                Array3f([1.0, 0.0], [0.0, 0.0], [0.0, 1.0])))
scene.build()
img = scene.render(ray, rd.SurfelRenderOptions.rgb())
print(img.rgb, img.alpha)   # [[0.6, 0, 0.24]] [0.84]

SurfelAppearance.sh(opacity, coeffs, sh_degree) uses spherical-harmonic colors, and SurfelAppearance.features(opacity, values, channel_count) renders arbitrary feature channels with SurfelRenderOptions.feature(channel_count). update_geometry and update_appearance replace either half on a built scene.

In Torch, SurfelScene offers intersect, visible, trace_reflections, composite_alpha, and transmittance; rendering is Dr.Jit only.

Mixed scenes#

MixedScene puts meshes, SDF grids, and surfel clouds into one scene. A query returns the nearest hit over all of them.

mixed = rd.MixedScene()
mixed.add_mesh(rd.Mesh(Array3f([-1, 1, 1, -1], [-1, -1, 1, 1], [-1.0] * 4),
                       Array3i([0, 0], [1, 2], [2, 3])))
mixed.add_sdf(sphere)
mixed.add_surfel(rd.SurfelCloud(Array3f([0.0], [0.0], [1.5]), Array3f([0.3], [0.0], [0.0]),
                                Array3f([0.0], [0.3], [0.0]), Float([0.8])))
mixed.build()
 
rays = rd.Ray(Array3f([0.0, 0.9], [0.0, 0.9], [3.0, 3.0]), Array3f([0.0, 0.0], [0.0, 0.0], [-1.0, -1.0]))
hit = mixed.intersect(rays)
print(hit.t, hit.shape_id)          # [1.5, 4] [2, 0]: the surfel, then the floor
print(mixed.transmittance(rays))    # [0, 0]

MixedScene answers intersect, visible, trace_reflections, and transmittance. It has no edge or diffraction queries: SDFs and surfels have no edges. The Torch MixedScene also has add_sdf_batch.

Related#