Skip to content

README.md

Latest commit

 

History

History
387 lines (292 loc) · 22.3 KB

README.md

File metadata and controls

387 lines (292 loc) · 22.3 KB
Raw

SyncLight Dataset

This folder contains the tools for the dataset used to train and evaluate SyncLight. The data itself is hosted on the Hugging Face Hub at davidserra9/synclight (CC BY 4.0).

The dataset is released as one image per light source: every camera of every scene was rendered (or photographed) once for each light, with only that light switched on. Light is additive, so a weighted sum of these per-light images gives the scene under any combination of lights, intensities and colours. Instead of a fixed set of image pairs, you download the per-light images and generate as many relighting pairs as you need with generate_pairs.py. The provided configs reproduce the distribution and scale of the pairs used in the paper: about 1 M training pairs.

Source Content Per-light images Scenes (train / test) Download
Infinigen procedurally generated rooms (bathroom, bedroom, dining room, kitchen, living room) linear HDR renders, EXR float32 RGB, 1280×720 320 / 9 223 GB / 6.9 GB
BlenderKit artist-made indoor and outdoor scenes linear HDR renders, EXR float32 RGB, 1280×720 37 / 3 6.5 GB / 0.5 GB
Real multi-view photographs of real rooms (two cameras) RAW photographs, DNG / CR2 35 / 4 4.8 GB / 0.7 GB
Total 22,678 EXR + 303 RAW 392 / 16 242 GB

Contents:

  1. Install
  2. Download
  3. What is in a scene
  4. Generate training pairs
  5. Output format
  6. How the generator works
  7. Customising the pairs
  8. Reproducing the paper's data
  9. Speed and disk space
  10. Using the pairs with SyncLight

1. Install

From the repository root, install either the full package plus the dataset extras:

pip install -e ".[dataset]"

or only what the generator needs (no PyTorch):

pip install -U huggingface_hub numpy opencv-python pillow pyyaml scipy "OpenEXR>=3.3" rawpy

OpenEXR reads the synthetic renders and rawpy reads the real RAW photographs; you only need the one for the sources you use. All commands below are run from the repository root.

2. Download

download.sh downloads the per-scene tars with hf download and extracts them:

bash dataset/download.sh data/raw                    # everything (242 GB)
bash dataset/download.sh data/raw blenderkit         # one source, both splits
bash dataset/download.sh data/raw real test          # one source, one split (0.7 GB)
  • Resuming: tars are kept in data/raw/.tars, so an interrupted download resumes where it stopped.
  • Disk space: delete data/raw/.tars after extraction, or run with REMOVE_TARS=1 to delete each tar as soon as it is extracted.
  • Manual download: you can also download tars directly, e.g. hf download davidserra9/synclight --repo-type dataset --include "blenderkit/test/*" --local-dir ..., and extract them with tar -xf.

The extracted layout:

data/raw/
├── infinigen/{train,test}/<room>/<seed>/rendered/
├── blenderkit/{train,test}/<scene>/rendered/
└── real/{train,test}/<scene>/captured/

The test folders hold the held-out scenes used for evaluation in the paper; no scene appears in both splits. Scenes without any usable camera pair were left out of the release (see metadata/skipped_scenes.jsonl on the Hub); metadata/manifest.jsonl lists every tar with its size and SHA-256.

3. What is in a scene

Synthetic scenes (Infinigen, BlenderKit), in rendered/:

File Content
camera_<rig>_<cam>_<light>_<vis>.exr the scene seen by camera <cam> of rig <rig>, lit only by light <light>; linear radiance, channels R, G, B (float32)
camera_<rig>_<cam>_<light>_light_map.png where light <light> appears in that camera's image (bright blob on black)
camera_transforms.json position, rotation and intrinsics of every camera, per rig

Real scenes, in captured/:

File Content
scene<N>_<cam>_<light>_<vis>.{DNG,CR2} RAW photograph from camera position <cam> with only light <light> on
scene<N>_<cam>_<light>_<vis>_light_map.png where that light appears in the photograph

The cameras of a rig look at the same part of the scene from different viewpoints. Real scenes have one rig and no calibrated poses.

<vis> says from where a light can be seen, which decides what can be edited:

<vis> Meaning Editable
camvis visible from this camera yes: edits are drawn on it in the lightmap
rigvis visible only from other cameras of the rig no, but it is switched on and off at random
nonvis (synthetic) / ambient (real) not visible from any camera: ambient fill or window light no

The EXRs keep only the final image (Combined pass) of the original Blender renders, with the pixel values unchanged. The Blender scene files are not included: Infinigen scenes can be regenerated with Infinigen, and BlenderKit assets are covered by BlenderKit's own licences.

4. Generate training pairs

Use one config per source; each matches the settings of the paper's pairs:

python dataset/generate_pairs.py --config dataset/configs/infinigen.yaml \
    --input data/raw/infinigen/train --output data/pairs/infinigen/train --workers 16

python dataset/generate_pairs.py --config dataset/configs/blenderkit.yaml \
    --input data/raw/blenderkit/train --output data/pairs/blenderkit/train --workers 16

python dataset/generate_pairs.py --config dataset/configs/real.yaml \
    --input data/raw/real/train --output data/pairs/real/train --workers 4

Repeat with test for the evaluation scenes.

  • Workers: real captures need about 3 GB of RAM per worker, synthetic scenes under 0.5 GB.
  • Resuming: scenes that already have a transforms.json are skipped, so an interrupted run resumes with the same command. Add --overwrite to regenerate them.
  • Subsets: --scenes Bathroom/10 Kitchen/5 limits the run to some scenes.
  • Counting first: --dry-run counts the pairs and images a config would produce, without rendering anything. It only reads file names, so it is instant:
python dataset/generate_pairs.py --config dataset/configs/blenderkit.yaml --input data/raw/blenderkit/train --dry-run

5. Output format

The output mirrors the input scene folders, with one folder of pairs per scene: png_simple/ for Infinigen and png_blended/ for BlenderKit and real.

data/pairs/blenderkit/train/<scene>/png_blended/
├── cam_<rig>_<cam>_<idx>_0_guide.png        reference view, before the edit
├── cam_<rig>_<cam>_<idx>_1_guide.png        reference view, after the edit
├── cam_<rig>_<other>_<idx>_0.png            another view of the same rig, before
├── cam_<rig>_<other>_<idx>_1.png            the same view, after
├── cam_<rig>_<cam>_<idx>_0_to_1_lightmap.png   the edit, drawn in the reference view
├── cam_<rig>_<cam>_<idx>_1_to_0_lightmap.png   the reverse edit (when stored)
├── transforms.json                          list of all pairs
└── generation_config.yaml                   the exact config used, for reproducibility

transforms.json lists one entry per training pair:

{
  "images": [
    {
      "input_guide":  "cam_0_1_00000_0_guide.png",
      "target_guide": "cam_0_1_00000_1_guide.png",
      "input_image":  "cam_0_2_00000_0.png",
      "target":       "cam_0_2_00000_1.png",
      "lightmap":     "cam_0_1_00000_0_to_1_lightmap.png",
      "relative_rotation":    [0.03, 0.80, 0.08],
      "relative_translation": [12.37, 0.45, -6.52],
      "intrinsics": {"focal_length_px": 711.1, "cx": 640.0, "cy": 360.0, "resolution": [1280, 720], "...": "..."}
    }
  ],
  "cameras": [{"0": ["0", "1", "2", "3"]}]
}
  • Poses: the relative pose of the other view with respect to the reference view is given as XYZ Euler angles plus a translation. In the real split the pose and intrinsics fields are zeros, because those captures are not calibrated. SyncLight itself does not use camera poses.
  • Reverse pairs: a pair and its reverse share the same images with input and target swapped, and use the _1_to_0_ lightmap.

Lightmaps

A lightmap is a 4-channel float map in [−1, 1] that is non-zero only on the pixels of the edited light:

Channel Meaning
0: activation −1 turn the light off · 0 no change · 1 set the light to the values below
1: L target intensity, intensity = (L + 1) / 2
2: a target colour, CIELAB a / 128
3: b target colour, CIELAB b / 128

They are stored as 16-bit RGBA PNGs, with each value x encoded as round(x · 32767) + 32768. −1, 0 and 1 are exact, and every other value is within 1.6·10⁻⁵ of the original float. Read them with the helper in the package:

from src.synclight.lightmap_io import load_lightmap
lm = load_lightmap("cam_0_1_00000_0_to_1_lightmap.png")   # float32 (H, W, 4)

Image viewers show these PNGs as semi-transparent because channel b is stored in alpha. That is expected. Set output.lightmap: npy to write float32 .npy files instead. test.py and the Gradio demo accept both formats.

6. How the generator works

1. Light states. A state gives every light an intensity in [0, 1] and an RGB colour. The image of a state is

image = tonemap( exposure · Σ_i  intensity_i · colour_i · light_i )

where light_i is the per-light HDR image of light i.

2. Samples. A sample is a before state and an after state that differ in exactly one camvis light: the edit. For every camera of a rig, used as the reference view, two families of samples are generated:

  • Solo sweep (solo_sweep). Only when the scene has an ambient light. The before state is the ambient light alone. In the after state, one visible light is switched on in each of 8 saturated hues (red, orange, yellow, green, cyan, blue, purple, magenta) and in n_temperatures colour temperatures. These samples teach clean colour edits.
  • Random combinations (pairs.per_light samples per editable light):
    • Before state: each ambient light is on with probability lights.p_ambient_on and each visible light with probability lights.p_visible_on, at an intensity drawn from lights.on_intensity. All lights share one colour drawn from colors.scene.
    • After state: one edit to the target light, chosen with the edits weights:
      • toggle: on → off, or off → on at full intensity
      • intensity: a bright light is dimmed to dim_to, a dim one brightened to brighten_to, an off one partially switched on to partial_on
      • color: the light gets a new colour drawn from colors.edit, and is switched on if it was off

3. Colours. Every colour draw is white, a colour temperature or a saturated hue, with the probabilities in colors.scene (the shared colour of the before state) or colors.edit (the new colour of a colour edit):

  • Temperatures come from the colors.temperatures list (Kelvin), or uniformly from colors.temperature_range.
  • Saturated hues depend on colors.vivid.mode:
    • named: the 8 fixed hues
    • jitter: random hue, saturation and value around each of the 8 hues
    • any: any hue, with the saturation and value ranges you set
  • Per-light colours: colors.per_light: true gives every light its own colour.

4. Views and pairs. Each sample is rendered in the reference view (the guide images and the lightmap) and in up to pairs.target_views other cameras of the rig. Each other view gives one training pair, plus its reverse (after → before) when the sample stores it. Per reference camera, with k editable lights in a rig of C cameras:

samples = k · (8 + n_temperatures)   [if the scene has an ambient light]   +   k · per_light
pairs   = Σ over samples of (C − 1) · (1 + reverse)

pairs.max_per_scene drops samples at random, keeping the same fraction in every camera, until the scene fits under the cap.

5. Tone mapping. Linear images are multiplied by tonemap.exposure, mapped to [0, 1] with tonemap.operator, gamma-encoded and quantised to 8 bits:

  • Operators: reinhard x/(1+x), reinhard_white, aces, hable (filmic) or clip.
  • Auto exposure: tonemap.auto_exposure: {percentile: 99, target: 0.9} picks the exposure per sample from the reference before image. It then uses that exposure for every image of the sample, so before and after stay photometrically consistent.

6. Lightmaps. The edited light's pixels are found by thresholding its light-position map at output.lightmap_threshold, and filled with the values described in Lightmaps.

7. Determinism. Every reference camera has its own random generator, seeded from seed, the scene name, the rig and the camera. The same config gives the same pairs regardless of the number of workers, and --dry-run counts exactly what a real run produces. Change seed to get a different set of pairs from the same distribution.

7. Customising the pairs

Any config value can be overridden with --set key=value (values are parsed as YAML), or you can copy a config and edit it.

Colour palettes

colors.palette applies a preset on top of a config. Any colors.* value you set with --set still overrides it.

Palette Scene colour (white / temperature / saturated) Colour edits Notes
neutral 50 / 50 / 0 20% white, 80% temperature 3000–6500 K
warm 20 / 80 / 0 temperature only 2200–4000 K (tungsten)
cool 20 / 80 / 0 temperature only 5000–9000 K (daylight, fluorescent)
colorful 40 / 30 / 30 40% temperature, 60% saturated hues jittered around the 8 named ones
wild 10 / 20 / 70 90% saturated any hue; every light gets its own colour

Recipes

G="python dataset/generate_pairs.py --input data/raw/blenderkit/train --output data/pairs_custom/blenderkit/train"

# twice as many random samples, warm lights only
$G --config dataset/configs/blenderkit.yaml --set pairs.per_light=20 colors.palette=warm

# colour edits only, any hue, every light its own colour
$G --config dataset/configs/blenderkit.yaml --set colors.palette=wild edits.toggle=0 edits.intensity=0 edits.color=1

# filmic tone mapping with automatic exposure
$G --config dataset/configs/blenderkit.yaml --set tonemap.operator=aces "tonemap.auto_exposure={percentile: 99, target: 0.9}"

# fewer pairs: 2 other views per sample, no reverse pairs, no solo sweep
$G --config dataset/configs/blenderkit.yaml --set pairs.target_views=2 pairs.reverse=false solo_sweep.enabled=false

# smaller and much faster real-capture pairs
python dataset/generate_pairs.py --config dataset/configs/real.yaml --input data/raw/real/train \
    --output data/pairs_small/real/train --set output.max_side=768 output.resize_inputs=true

Parameter reference

Key Default Meaning
source infinigen infinigen, blenderkit or real: how base files are found and read
output_subdir png_simple folder written inside each output scene
seed 0 master seed
pairs.per_light 5 random-combination samples per editable light, per reference camera
pairs.max_per_scene null cap on pairs per scene
pairs.reverse true also store the reverse edit of random samples
pairs.target_views all other views rendered per sample (all or a number)
pairs.min_cameras 2 rigs with fewer cameras are skipped
solo_sweep.enabled true add the solo colour sweep
solo_sweep.vivid 8 hues hues of the sweep
solo_sweep.n_temperatures 5 colour temperatures added to the sweep, drawn from solo_sweep.temperatures
solo_sweep.intensity 1.0 intensity of the swept light
solo_sweep.reverse_first 3 only the first N sweep colours also store the reverse edit
lights.p_ambient_on 0.8 probability that each ambient light is on in the before state
lights.p_visible_on 0.6 probability that each visible light is on in the before state
lights.on_intensity [0.4, 1.0] intensity range of lights that are on
edits.toggle / intensity / color 0.7 / 0.3 / 0 relative weights of the edit types
edits.dim_to, brighten_to, partial_on [0.2, 0.5], [0.7, 1], [0.5, 1] target intensity ranges of intensity edits
edits.bright_threshold 0.6 lights above it are dimmed, below it brightened
colors.palette null preset (see above)
colors.scene 0.7 / 0.18 / 0.12 white / temperature / saturated weights of the shared scene colour
colors.edit 0 / 0.7 / 0.3 weights of the new colour in colour edits
colors.per_light false every light draws its own colour
colors.temperatures list colour temperatures to choose from (K)
colors.temperature_range null continuous temperature range instead of the list
colors.vivid.mode jitter named, jitter or any
colors.vivid.names 8 hues hues used by named and jitter
colors.vivid.saturation, value [0.7, 1], [0.7, 1] ranges used by any
tonemap.operator reinhard reinhard, reinhard_white, aces, hable, clip
tonemap.white 4.0 white point of reinhard_white
tonemap.exposure 1.0 multiplier on linear radiance
tonemap.auto_exposure null {percentile, target}: exposure per sample from the reference before image
tonemap.gamma 2.2 display gamma, or srgb
tonemap.quantize floor floor (as in the paper's data) or round
output.max_side null downscale outputs so the longest side is at most this many pixels
output.resize_inputs false downscale base images before mixing instead of after tone mapping: about 6× faster for RAW, with a mean difference under 1 grey level (up to ~40 on small, very bright spots)
output.lightmap png16 png16 or npy
output.lightmap_size image image (same size as the images) or native (size of the light-position maps)
output.lightmap_threshold 0.098 light-position map value above which a pixel belongs to the light
output.precision float32 float64 reproduces the original renderer bit for bit

The defaults in this table are those of the script itself. The configs in configs/ override some of them per source.

8. Reproducing the paper's data

Config Random samples per light Pair cap per scene Edits (switch / intensity / colour) Tone mapping Output size
infinigen.yaml 5 5,000 70 / 30 / 0 Reinhard, γ 2.2 1280×720
blenderkit.yaml 10 none 55 / 25 / 20 Reinhard, γ 2.2 1280×720
real.yaml 10 none 55 / 25 / 20 clip, exposure 1.0, γ 2.2 longest side 1280

These configs use the same sampling rules, colour lists, tone mapping and light-map thresholds as the scripts that produced the paper's pairs. Those original runs were not seeded, so a new run produces different pairs with the same distribution and practically the same size:

Source / split Pairs (paper) Pairs (configs) Images (paper) Images (configs)
Infinigen train 898,267 920,878 1,579,444 1,617,594
Infinigen test 27,236 29,516 47,788 51,572
BlenderKit train 44,326 44,420 74,822 74,950
BlenderKit test 3,198 3,222 5,464 5,496
Real train 12,718 12,756 22,652 22,700
Real test 1,718 1,712 3,066 3,058

The renderer is exact. With --set output.precision=float64, the deterministic samples (the start of each solo sweep) are bit-identical to the paper's images and lightmaps for all three sources. The default float32 is faster and differs by at most one grey level.

9. Speed and disk space

Measured on one CPU core (--workers 1) with the paper configs, reading the per-light images from a local SSD:

Scene Pairs Images Time Output
Infinigen Bathroom/1000 (test) 804 1,580 252 s 1.1 GB
BlenderKit bare_room (test) 1,728 2,944 477 s 1.6 GB
Real scene15 (test) 312 536 907 s 0.8 GB
Real scene15 (test), output.resize_inputs=true 312 536 138 s 0.8 GB

Scaled to the full training splits (estimates):

Split Images CPU time With --workers 16 Disk
Infinigen train 1.62 M ~72 core-hours ~4.5 h ~1.1 TB
BlenderKit train 75 K ~3.4 core-hours ~15 min ~45 GB
Real train (exact / resize_inputs) 23 K ~11 / ~1.6 core-hours ~40 / ~6 min ~37 GB

Rendering costs about 0.16 s per image for EXR scenes. To make the output smaller and faster to produce, lower pairs.per_light, pairs.max_per_scene or pairs.target_views, or set output.max_side.

10. Using the pairs with SyncLight

A generated pair can be fed straight to the inference script. Pass the reference view and other views before the edit, plus the lightmap; compare the outputs with the _1 images:

P=data/pairs/blenderkit/test/bare_room/png_blended
python test.py \
  --image_paths $P/cam_0_0_00000_0_guide.png $P/cam_0_1_00000_0.png $P/cam_0_2_00000_0.png \
  --lightmap_path $P/cam_0_0_00000_0_to_1_lightmap.png \
  --output_dir outputs/bare_room_00000

To iterate over pairs in your own code:

import json, os
from PIL import Image
from src.synclight.lightmap_io import load_lightmap

scene = "data/pairs/blenderkit/train/abandoned_apartment/png_blended"
pairs = json.load(open(os.path.join(scene, "transforms.json")))["images"]
for p in pairs:
    guide_before = Image.open(os.path.join(scene, p["input_guide"]))
    guide_after  = Image.open(os.path.join(scene, p["target_guide"]))
    view_before  = Image.open(os.path.join(scene, p["input_image"]))
    view_after   = Image.open(os.path.join(scene, p["target"]))
    lightmap     = load_lightmap(os.path.join(scene, p["lightmap"]))   # (H, W, 4) float32

License

The dataset is released under CC BY 4.0. BlenderKit assets are subject to BlenderKit's own licences; the dataset contains renders only, not the assets. The code in this folder is released under the repository's Apache 2.0 licence.