Relight: relightable images (RTI) (C++ and Javascript)

Relight is a library to create and view (on the web) relightable images (RTI),

It supports Polinomial Texture Mapping (PTM), Hemispherical Harmonics (HSH) along with a new Principal Component Analysis (PCA) + Radial Basis Function (RBF) based algorithm with better fidelity (for the same cost and space).

It also supports RTI (and image) streamed tiled viewing based on DeepZoom, Google, Zoomify, IIP formats.

An online Web3d presentation of the paper A Compact Representation of Relightable Images for the Web presented at the Web3D 2018 conference might be of help

The package is composed of:

Rbf comparison

Bottom of a bottle
Courtesy of Musee d'Histoire de Marseille

Rbf comparison

PTM, HSH, RBF comparison
Courtesy of Palazzo Blu

Rbf comparison

HSH, RBF, YCC comparison
Courtesy of Palazzo Blu

Rbf comparison

All comparisons

Download

Source code is available at: https://github.com/cnr-isti-vclab/relight

RTI supported formats

PTM: Polynomial Texture Mapping

18 planes, or 9 using the LPTM where only the luminance change

HSH: Hemispherical Harmonics

27 planes, (9 per channel)

DMD: Discrete modal decomposition

Described in Discrete Modal Decomposition for surface appearance modelling and rendering, it is not supported.

PCA + RBF

Variable number of planes using a sampled basis and radial basis function interpolation.

Named as RBF<nplanes> such as RBF18.

PCA + RBF + YCC

Same algorithm as before, but using the Jpeg color space separately for each channel, resulting in a better (adjustable) chroma fidelity. YCC<nluma><nchroma><nchroma> such as YCC1044 for 10 luminance planes and 4+4 for the chroma.

BILINEAR

Bilinear interpolation replace radial basis function in the previous algorithms. Especially good for uneven sampling of the light directions.

Relight fitter

relight is a Qt command-line program to process a stack of photos into an RTI.

	Usage: relight [-frqpsScCey] <input folder> [output folder]

	input folder containing a .lp with number of photos and light directions
	optional output folder (default is ./)

	-b : rbf(default), ptm, lptm, hsh, yrbf, bilinear
	-p   : number of planes (default: 9)
	-y   : number of Y planes in YCC
	-r   : side of the basis bilinear grid (default 8)
	-q   : jpeg quality (default: 90)
	-s   : sampling rate for pca (default 4)
	-S   : sigma in rgf gaussian interpolation default 0.125 (~100 img)
	-C        : apply chroma subsampling 
	-e        : evaluate reconstruction error (default: false)

relight can also be used to convert .ptm files into relight format:

	relight [-q] <file.ptm> [output folder]
	-q   : jpeg quality (default: 90)
libvips can be used to [generate deepzoom, zoomify and google pyramidal formats](http://libvips.blogspot.com/2013/03/making-deepzoom-zoomify-and-google-maps.html), scripts can be found in the directory test.

relight.js

relight.js is a small Javascript library to render the RTI on a WebGL canvas. It support a variety of tiled formats.

To inizialize the library:

var relight = new Relight(canvas, options);

Canvas is either a css selector or the canvas object

Options is an object with the following keys:

  • url: path to a directory containing the web format
  • layout: deepzoom, zoomify, google, iip, iiif or image for the web format
  • path: used for iip layout, absolute path of the dir.
  • light: initial light, array x, y, z.
  • pos: initial view object { x: y: z:}
  • border: for tiled formats amount of prefetching tiles arount the view (default 1)
  • fit: scale and center the view on load

Members:

  • pos: the position of the view as {x: y: z: t} where:
    x and y are the coords of the center of the screen in full scale image.
    z is the zoom level with 0 being the original image.
    t is for interpolation in ms
  • light: array of x, y, z light direction
  • onload: function to be called when rendering is ready

Methods:

  • setUrl(url): change url
  • resize(width, height): resize canvas
  • zoom(amount, dt):
  • pan(dx, dy, dt): change the center of the view
  • setPosition(x, y, z, dt):
  • center(dt): center view (but does not change zoom
  • centerAndScale(dt): fit the view to the canvas
  • setLight(x, y, z): change light direction
  • draw(time): draw the canvas, use time for interpolation
  • redraw(): schedule an animaterequest

relight-viewer.js

It's an RTI (and images) viewer based on the relight.js library, and requires a .css file and a set of icons in the skin directory.

<!doctype html>
<html>
  <head>
    <link  rel="stylesheet" href="relight.css">
    <style>
      html, body, .relight { width:100%; height:100%; padding:0px; margin:0px; }
    </style>
    <script src="js/hammer.min.js"></script>
    <script src="relight-viewer.min.js"></script>
  </head>
<body>
  <div class="relight"></div>
  <script>
var Relight = new RelightViewer('.relight', { url: "rti-folder", layout: "deepzoom" });
  </script>
</body>
</html>

It takes the relight.js options and additionally:

  • pandelay: in ms the inertia of the movement (default 50)
  • zoomdelay: in ms the inertia of the zooming process (default 200)
  • zoomstep: amount of zoom for buttons and wheel (default: 0.25, meaning to 4 clicks to double the size)
  • lightsize: percentage of the viewer area used for light direction change (default 0.8)
  • scale: the size in mm of a pixel, to show the scale of the object (default 0, meaning disabled)

Relight format

Relight format is a directory containing an info.json file and a few images.

The JSON structure contains the following fields:

  • width:
  • height:
  • format: image extension: .jpg, .png etc.
  • type: one of lptm, hsh, rbf, bilinear ycc
  • nplanes: numer of coefficient planes
  • resolution: for bilinear types the side of the bilinear grid
  • colorspace: one of lrgb, rgb, ycc, mrgb, mycc
  • sigma: rbf interpolation parameter
  • lights: mandatory for rbf interpolation
  • materials: an array of materials, each material specify scale, bias and range
  • scale: an array of floats
  • bias: an array of floats
  • range: an array of floats
  • basis: an array of unsigned chars containing the basis for rbf, bilinear and ycc basis.

Each image contains 3 coefficient planes

  • PTM: r, g, b, 1, u, v, u^2, uv, v^2
  • HSH: see source code :)
  • RBF: PCA basis coefficients
  • BILINEAR: PCA basis coefficients
  • YCC: PCA basis coefficients as Y0C0C0, Y1C1C1... YkCkCk, Yk+1Yk+2Yk+3

Scale and bias will be applied to the texture coefficients (from [0, 1]) as: (c - bias)*scale