Relight:

A Compact Representation
of Relightable Images
for the Web

Federico Ponchio, Massimiliano Corsini, Roberto Scopigno

Relightable Images

Camel

Reflectance
Transformation
  Imaging

• Fixed view point
• Arbitrary lighting
• In practice directional light

Reflectance Transformation Imaging (RTI)

[R, G, B] = f(x, y, θ, φ)

Sampling the function

Light dome

Manual light positioning

A stack of images

~50-100 images  ~500MB!

Challenges

• Interpolation/Fitting
• Compact representation
• Efficient rendering

Sampled function*

*Actually one function per channel RGB

Continuous function*

*Which interpolation/approximation?

Polynomial fitting

Polynomial Texture Map (2001)
(PTM)

R(l) = r0 + r1lx + r2ly + r3lx2 + r4ly2 + r5lxly

G(l) = g0 + g1lx + g2ly + g3lx2 + g4ly2 + g5lxly

B(l) = b0 + b1lx + b2ly + b3lx2 + b4ly2 + b5lxly

Requires 18 coefficients

LPTM

Y(l) = y0 + y1lx + y2ly + y3lx2 + y4ly2 + y5lxly

R(l) = r0Y(l)

G(l) = g0Y(l)

B(l) = b0Y(l)

Requires 9 coefficients

It's a weighted sum of a basis

Spherical Harmonics

vs.

HemiSpherical Harmonics (HSH)

HSH light coefficients

H0 = 1 / sqrt(2π)
H1 = sqrt(6/π)(cos(φ) sqrt(cos(θ)-cos(θ)2))
H2 = sqrt(3/(2π))(2cos(θ)-1)
H3 = sqrt(6/π)(sqrt(cos(θ) - cos(θ)2)sin(φ))
H4 = sqrt(30/π)(cos(2φ)(-cos(θ) + cos(θ)2))
H5 = sqrt(30/π)(cos(φ)(2cos(θ)-1)sqrt(cos(θ) - cos(θ)2))
H6 = sqrt(5/(2π))(1 - 6cos(θ) + 6cos(θ)2)
H7 = sqrt(30/π)((2cos(θ)-1)sqrt(cos(θ) - cos(θ)2)sin(φ))
H8 = sqrt(30/π)((-cos(θ) + cos(θ)2)sin(2*φ))

HSH

R(l) = r0H0(l) + ... + r8H8(l)

G(l) = g0H0(l) + ... + g8H8(l)

B(l) = b0H0(l) + ... + b8H8(l)

Requires 27 coefficients

RTI Compression

• Coefficients scaled and quantized (8 bits)
• JPEG encoding

RTI Rendering

Texture lookup, dequantization, light coefficients.

uniform sampler2D planes[6];
uniform float light[6], bias[6], scale[6];
varying vec2 v_texcoord;

void main(void) {
	vec3 color = vec3(0);
	for(int j = 0; j < 6; j++) {
		vec4 c = texture2D(planes[j], v_texcoord);
		color.x += light[j]*(c.x - bias[j*3+0])*scale[j*3+0]; 
		color.y += light[j]*(c.y - bias[j*3+1])*scale[j*3+1]; 
		color.z += light[j]*(c.z - bias[j*3+2])*scale[j*3+2]; 
	}
	gl_FragColor = vec4(color, 1.0);
};

Used mostly in Cultural Heritage

• Inscriptions
• Cuneiform tablets
• Manuscripts
• Coins
• Small manufacts

Cheap, captures well the appearance

Both an inspection and a presentation tool

CHI

• Training, consulting.
• RTIBuilder
• RTIViewer

http://culturalheritageimaging.org/Technologies/RTI/

Custom Imaging

• RTI Card Importer
• RTI Processor
• PTM Fitter

https://custom-imaging.co.uk/software/

Web RTI Viewer

• Tiled approach
• Supports .PTM and .HSH
• Open source

http://vcg.isti.cnr.it/rti/webviewer.php

dhlab-basel/rti.js

• IIIF compliant
• Supports .PTM
• three.js
• Open source

https://github.com/dhlab-basel/rti.js

Improving quality/size ratio

Radial Basis Function (RBF)

Multispectral RTI analysis of heterogeneous artworks, Giachetti et.al.

RBF

• Very good RTI fidelity
• Requires ALL the original images!

RBF (approximated)

We need to approximate the array [ρ₁ ... ρ_N]
for each pixel.

Principal component analysis (PCA)

Distributive property!

Same computational cost

• Compute light coefficients once
  (this time using a small vector base)
• Weighted sum for each pixel.

A custom base using PCA

and PCA neglected the color red.

Luminance-Chroma decoupling

Convert images to YCbCr

Luminance vs Croma fidelity

Custom sampled basis

What if the light direction is not constant in our sampling?

or we compute a custom base in some other way?

Bilinear interpolation

The function is resampled with over a grid (9x9)
and costant for all pixels.

Half sphere to square?

Flattened octahedron projection

Bilinear resampling (1)

We need to find 81 values such that when interpolated the surface is as closed as possible to the original samples

• N original samples ρ_i
• M grid resamples η_j

Minimize |Aη - ρ|² where A is the bilinear interpolation matrix

Bilinear resampling (2)

Tikhonov regularization, since we have a prior ( and N < M happens)

Rρ = η₀   ( R is the RBF coefficient matrix)

Minimize |ρ − Aη|² + τ|η − η₀|²

Then build a basis using PCA

or any other algorithm...

Same rendering algorithm

Bonus: texture access with bilinear interpolation

WebGL rendering changes

Custom base is delivered in JSON (~5-40Kb)

Light dependent coefficients computed in Javascript

Conclusions

• Better quality/size ratio
• Same computational cost (same opengl code!)
• Luma-chroma optimization
• Flexible: any sampled base.

Relight on github

https://github.com/cnr-isti-vclab/relight

RTI encoder supports PTM, HSH, RBF, YCC and BILINEAR

JS Viewer support multiresolution: deepzoom, zoomify, google, iip...