of Relightable Images

for the Web

Federico Ponchio, Massimiliano Corsini, Roberto Scopigno

Reflectance

Transformation

Imaging

- Fixed view point
- Arbitrary lighting
- In practice directional light

Reflectance Transformation Imaging (RTI)

[R, G, B] = f(x, y, θ, φ)

Light dome

Manual light positioning

~50-100 images ~500MB!

All the pixels for one light

All the lights for one pixel

(a column in the stack)

- Interpolation/Fitting
- Compact representation
- Efficient rendering

*Actually one function per channel RGB

*Which interpolation/approximation?

(PTM)

`R(l) = r`

_{0} + r_{1}l_{x} + r_{2}l_{y} +
r_{3}l_{x}^{2} + r_{4}l_{y}^{2} +
r_{5}l_{x}l_{y}

`G(l) = g`

_{0} + g_{1}l_{x} + g_{2}l_{y} +
g_{3}l_{x}^{2} + g_{4}l_{y}^{2} +
g_{5}l_{x}l_{y}

`B(l) = b`

_{0} + b_{1}l_{x} + b_{2}l_{y} +
b_{3}l_{x}^{2} + b_{4}l_{y}^{2} +
b_{5}l_{x}l_{y}

Requires 18 coefficients

`Y(l) = y`

_{0} + y_{1}l_{x} + y_{2}l_{y} +
y_{3}l_{x}^{2} + y_{4}l_{y}^{2} +
y_{5}l_{x}l_{y}

`R(l) = r`

_{0}Y(l)

`G(l) = g`

_{0}Y(l)

`B(l) = b`

_{0}Y(l)

Requires 9 coefficients

vs.

```
H
```

_{0} = 1 / sqrt(2π)

H_{1} = sqrt(6/π)(cos(φ) sqrt(cos(θ)-cos(θ)^{2}))

H_{2} = sqrt(3/(2π))(2cos(θ)-1)

H_{3} = sqrt(6/π)(sqrt(cos(θ) - cos(θ)^{2})sin(φ))

H_{4} = sqrt(30/π)(cos(2φ)(-cos(θ) + cos(θ)^{2}))

H_{5} = sqrt(30/π)(cos(φ)(2cos(θ)-1)sqrt(cos(θ) - cos(θ)^{2}))

H_{6} = sqrt(5/(2π))(1 - 6cos(θ) + 6cos(θ)^{2})

H_{7} = sqrt(30/π)((2cos(θ)-1)sqrt(cos(θ) - cos(θ)^{2})sin(φ))

H_{8} = sqrt(30/π)((-cos(θ) + cos(θ)^{2})sin(2*φ))

`R(l) = r`

_{0}H_{0}(l) + ... + r_{8}H_{8}(l)

`G(l) = g`

_{0}H_{0}(l) + ... + g_{8}H_{8}(l)

`B(l) = b`

_{0}H_{0}(l) + ... + b_{8}H_{8}(l)

Requires 27 coefficients

- Coefficients scaled and quantized (8 bits)
- JPEG encoding

RAW: 80MB

HSH: 3.0MB

PTM: 1.9MB

LPTM: 1.4MB

Texture lookup, dequantization, light coefficients.

```
uniform sampler2D planes[6];
uniform float light[6], bias[6], scale[6];
varying vec2 v_texcoord;
void main(void) {
vec3 color = vec3(0);
for(int j = 0; j < 6; j++) {
vec4 c = texture2D(planes[j], v_texcoord);
color.x += light[j]*(c.x - bias[j*3+0])*scale[j*3+0];
color.y += light[j]*(c.y - bias[j*3+1])*scale[j*3+1];
color.z += light[j]*(c.z - bias[j*3+2])*scale[j*3+2];
}
gl_FragColor = vec4(color, 1.0);
};
```

- Inscriptions
- Cuneiform tablets
- Manuscripts
- Coins
- Small manufacts

Cheap, captures well the appearance

Both an inspection and a presentation tool

- Training, consulting.
- RTIBuilder
- RTIViewer

- RTI Card Importer
- RTI Processor
- PTM Fitter

- Tiled approach
- Supports .PTM and .HSH
- Open source

- IIIF compliant
- Supports .PTM
- three.js
- Open source

Multispectral RTI analysis of heterogeneous artworks, Giachetti et.al.

- Very good RTI fidelity
- Requires ALL the original images!

We need to approximate the array [*ρ*_{1} ... *ρ*_{N}]

for each pixel.

Distributive property!

- Compute light coefficients once

(this time using a small vector base) - Weighted sum for each pixel.

A *custom* base using PCA

and PCA *neglected* the color red.

Convert images to YCbCr

What if the light direction is not constant in our sampling?

or we compute a custom base in some other way?

The function is resampled with over a grid (9x9)

and costant for all pixels.

Flattened octahedron projection

We need to find 81 values such that when interpolated the surface is as closed as possible to the original samples

- N original samples
*ρ*_{i} - M grid resamples
*η*_{j}

Minimize *|Aη - ρ| ^{2}* where

Tikhonov regularization, since we have a prior ( and N < M happens)

*Rρ = η _{0}* ( R is the RBF coefficient matrix)

Minimize *|ρ − Aη| ^{2} + τ|η − η_{0}|^{2}*

or any other algorithm...

Same rendering algorithm

Bonus: texture access with bilinear interpolation

PTM mse [0,25]

HSH

RBF

BILINEAR

Custom base is delivered in JSON (~5-40Kb)

Light dependent coefficients computed in Javascript

- Better quality/size ratio
- Same computational cost (same opengl code!)
- Luma-chroma optimization
- Flexible: any sampled base.

https://github.com/cnr-isti-vclab/relight

RTI encoder supports PTM, HSH, RBF, YCC and BILINEAR

JS Viewer support multiresolution: deepzoom, zoomify, google, iip...