# Photo Tourism, rebuilt An implementation of *Photo Tourism: Exploring Photo Collections in 3D* (Snavely, Seitz and Szeliski, SIGGRAPH 2006), written for the book *Computational Photography* by Frédo Durand. It has two halves that meet at one file: - `recon/` is structure from motion in Python. Photographs go in; out comes `scene.json`, with every camera's position, orientation and focal length, a sparse point cloud, and the tables the viewer needs. - `viewer/` is a WebGL explorer that navigates that scene the way the paper does. You click a camera to stand where its photograph was taken, then move between photographs using the paper's geometric relations, or drag a box around something to be taken to the photograph that shows it best. It also contains the same reconstruction written in JavaScript. `viewer/reconstruct.html` registers your own photographs inside the browser tab, without a server doing the work, and opens the result in the same explorer. One scene is included, the Brandenburg Gate, built from 100 photographs on Wikimedia Commons of which 79 register. You can explore it before installing anything except Python. ## Quick start git clone https://github.mit.edu/fredo/phototourism.git cd phototourism You need Python 3.10 or later. The explorer runs in any current browser. **1. Explore the included scene.** No packages needed. python3 viewer/serve.py Open http://127.0.0.1:8130/ . Drag to orbit, scroll to move closer, and click a camera frustum to fly into its photograph. From there, the panel lists the paper's moves: details, similar views, zoom-outs, a step left or right, and a step back. Press "select", then drag a box over part of the scene, and the viewer scores every camera on how well it shows that part and flies to the best one. `viewer/serve.py` is a static file server that turns caching off. Opening the HTML files straight from disk does not work, because a page loaded that way cannot fetch its scene. **2. Reconstruct your own photographs, in the browser.** Same server, no packages needed. Open http://127.0.0.1:8130/reconstruct.html and drop in photographs of one subject taken from different positions. Six to twenty is a good first try. The page finds keypoints, matches every pair across a pool of Web Workers, keeps the pairs that fit an epipolar geometry, joins the matches into tracks, places the cameras one at a time with bundle adjustment, and opens the result in the explorer. Your photographs are never uploaded: they are decoded and measured in the tab, and closing the tab discards everything. On a laptop, six photographs of the Middlebury temple take about 1.5 seconds, and ten overlapping photographs of the Brandenburg Gate place nine cameras in about 3 seconds. Matching is quadratic in the number of photographs, so forty take much longer than twenty. What decides success is overlap. The same page, given ten photographs of the Gate chosen without regard to overlap, found only 4 of 45 pairs that agreed on a geometry and could not start. When that happens the page says why and does not open the explorer. Take photographs of one subject a few steps apart, so that each overlaps its neighbours by half or more. **3. Reconstruct with the Python pipeline.** This is the more capable path: it estimates lens distortion, uses more features, and can use a learned matcher. python3 -m pip install -r requirements.txt python3 recon/build_scene.py path/to/your/photos myscene python3 viewer/serve.py Then open http://127.0.0.1:8130/?scene=myscene . The scene is written to `viewer/scenes/myscene/` and added to the picker in the explorer's header. Run the commands from the package's top directory: feature and match caches are written to `.cache/` there, which makes a second run on the same photographs much faster. ## Requirements Core, in `requirements.txt`: numpy, scipy, opencv-python (4.4 or later, for SIFT) and Pillow. Optional, in `requirements-optional.txt`, each needed only for the feature named: - `pycolmap`, to compare a reconstruction against COLMAP's (`recon/oracle.py`) or to load a COLMAP model as ground truth. - `torch` and `lightglue`, for the learned matcher (`--deep`, see below). - `websocket-client` and Google Chrome, for `viewer/sfm/check_page.py`, which drives the upload page in headless Chrome. Node.js 18 or later runs the JavaScript tests. Nothing in the viewer needs a build step: the two WebAssembly kernels are included prebuilt in `viewer/sfm/wasm/`. ## What is in the package recon/ the Python pipeline build_scene.py entry point: a folder of photographs in, a viewer scene out pipeline.py the stages joined up features.py SIFT, cached per image matching.py the ratio test in blocks, then a fundamental matrix per pair tracks.py union-find over the verified matches sfm.py the incremental reconstruction bundle.py bundle adjustment with an analytic Jacobian geometry.py eight-point, P3P, triangulation, the small linear algebra nav.py the paper's navigation tables export.py writes scene.json and the binary files beside it duplicate_check.py checks a finished scene for duplicated surfaces datasets.py reads a folder, a Middlebury set, or a COLMAP model fetch_commons.py collects a landmark's photographs from Wikimedia Commons deepmatch.py optional SuperPoint and LightGlue matching standalone.py packs a scene and the viewer into one HTML file test_*.py the tests, see below viewer/ the explorer, the upload page and their tests index.html, app.js the explorer render.js, scene.js WebGL drawing and the scene model reconstruct.html/.js the in-browser reconstruction page inspect.html replays how a scene was built, stage by stage serve.py a local server with caching turned off sfm/ the browser pipeline: detect, match, tracks, recon, bundle, nav, each ported from the Python and tested against it scenes/brandenburg/ the included scene halide/ the Halide generators for the two WebAssembly kernels LICENSE MIT for the code; the photographs keep their own licences docs/NOTES.md the project's lab notebook: every measurement and every approach that was tried and rejected, with the numbers docs/DESIGN.md the original design and milestones ## How the reconstruction works The pipeline follows the paper's Section 4. 1. Features. SIFT on a copy of each photograph scaled to at most 1,600 pixels. Photographs smaller than 1,100 pixels are upsampled first, which quadruples the keypoints on small images such as the Middlebury sets. 2. Matching. For every pair, the two nearest neighbours of each descriptor, kept when the nearest is clearly better than the second (ratio 0.75), in both directions. Then a fundamental matrix by the normalized eight-point algorithm inside RANSAC. Pairs with fewer than twenty inliers are dropped. 3. Tracks. Matches are joined into tracks, each meant to be one point in the world. A track that contains two keypoints from the same photograph is inconsistent and is discarded. 4. Seed. The pair with the most matches that a single homography cannot explain, which rules out pairs related by a pure rotation. 5. Growth. Cameras are added in batches by resection (P3P inside RANSAC), new points are triangulated, bundle adjustment runs, and observations that disagree are removed. Repeat. 6. Export. The final cloud is checked, the navigation tables are computed, and everything is written to `viewer/scenes//`. A few choices depart from the paper or from the obvious setting. Each was made on measurement, and `docs/NOTES.md` has the numbers. - New cameras are posed with P3P rather than the paper's six-point DLT. At minimal sample size the DLT gave a median rotation error of 6.3 degrees against P3P's 0.41, which is the difference between RANSAC finding a consensus and finding nothing. - The ratio test is 0.75, not a stricter 0.6. A stricter test discards correct matches between distant viewpoints faster than wrong ones, and the reconstruction falls apart. - Points seen by only two cameras are left out of bundle adjustment and re-triangulated afterwards. When a pair of photographs taken a second apart shares hundreds of points that nobody else sees, those points outvote the few observations that tie the pair to the rest of the scene. On the Statue of Liberty this produced a second, complete copy of the statue, tilted by 17.6 degrees, that fit its own observations better than the real one. - Before export, points are removed if their parallax in the final poses is below two degrees, or if they belong to a block that duplicates a surface already in the cloud. Camera convention, as in OpenCV and COLMAP: a world point X maps to the camera as x = R X + t, the camera looks down +z, and image y points down. ## The explorer This is the paper's Sections 5 and 6. - The cloud and one frustum per camera, with the photograph on its back face. - Click a frustum to fly into that photograph. - From a photograph, the six geometric moves. Details and zoom-outs come from how much of one photograph's points another one covers. Left and right fit the shared points as a zoom plus a shift and pick the neighbour whose horizontal shift is about a tenth of the frame. - Select a region and the viewer scores every camera by the paper's formula, E = E_visible + (1/3) E_angle + (2/3) E_detail, then flies to the winner. - Transitions use the paper's planar impostor: each photograph is drawn on a plane the two cameras share, through a camera that moves between them. Where the exporter measured a homography between the two photographs, that drives the transition instead. - "depth" (experimental, off by default) flies through each photograph's own estimated depth instead of a plane. It appears only for scenes that include depth maps, which the Brandenburg scene does. The code that makes depth maps is not in this package: it depends on large learned models. - "copy link" encodes the current view in the URL. `recon/standalone.py` packs a scene and the explorer into one HTML file that opens from disk with no server, which is the simplest way to send a reconstruction to someone: python3 recon/standalone.py brandenburg --out brandenburg.html ## Getting photographs - Your own. Walk around one subject and take photographs from many positions, with plenty of overlap between neighbours. Keep one focal length if you can: move closer rather than zooming. Plain walls, glass and repeated identical elements (windows, arches, chairs) are hard for any matcher. - A landmark from Wikimedia Commons, with every photograph's author and licence recorded in `credits.json`: python3 recon/fetch_commons.py "Brandenburg Gate" --n 100 --out data/brandenburg python3 recon/build_scene.py data/brandenburg `--survey` compares candidate landmarks before downloading. Commons categories mix photographs with drawings, souvenirs and other things that happen to share the name, so look at what arrived before spending time on matching. `docs/NOTES.md` has a section on the ways collection goes wrong. - The Middlebury multi-view stereo sets, which come with camera poses measured on a gantry, so `build_scene.py` reports the error of the reconstruction against them. Download templeRing or templeSparseRing from https://vision.middlebury.edu/mview/data/ and unpack it under `data/`. Cite Seitz et al. 2006 if you use them. - A COLMAP model (a folder with `cameras.bin`, `images.bin` and `points3D.bin` beside the images). `build_scene.py` recognises it and reports agreement with COLMAP's poses. `build_scene.py` decides what a folder is from what it contains, so none of these needs any configuration. Run it with no arguments to list what is under `data/`. ## Options python3 recon/build_scene.py [name] [--deep] [--rematch] `--deep` replaces SIFT with SuperPoint features and LightGlue matching (needs `torch` and `lightglue`). It helps most on large collections and on surfaces SIFT describes poorly: on the Statue of Liberty it registered 119 of 125 reachable photographs against SIFT's 89, and on a set of Sydney photographs 104 against 28. It is not always better. On the Middlebury temple, a repetitive stone facade, it registered 16 of 47 photographs where SIFT registered all 47. Try both. It is also much slower, about half a second per pair on Apple silicon. `--rematch` re-matches pairs of photographs that the first reconstruction places close together, with a looser ratio test, and keeps a new match only if a third photograph confirms it. ## Checking a reconstruction A reconstruction can be wrong while every number it reports looks fine. The duplicated statue mentioned above had a lower reprojection error than the rest of the scene. So check the cloud itself: python3 recon/duplicate_check.py myscene It reports three things and fails a scene on any of them: points whose parallax is below two degrees in the final poses; blocks of points seen by only one pair of photographs that sit at a different depth than the rest of the cloud along the same rays (a duplicated surface); and cameras whose points almost all come from a single partner photograph, which means they were attached to that photograph rather than to the scene. `build_scene.py` prints the same verdict at the end of every build. ## Tests python3 recon/test_synthetic.py # the solver on synthetic scenes with known answers python3 recon/test_bundle_jac.py # the analytic Jacobian against finite differences python3 recon/test_matching.py # the blocked matcher against a direct one python3 recon/test_track_ops.py # merging tracks, and holding two-view points out python3 recon/test_nav_step.py # the step-left and step-right rule python3 recon/test_imports.py # every module imports and a small scene builds node viewer/test_scene.js brandenburg # the viewer's arithmetic on a built scene The browser pipeline is tested against the Python it was ported from. Each test first needs a reference answer written by the Python: python3 viewer/sfm/make_sfm_ref.py && node viewer/sfm/test_recon.js && node viewer/sfm/test_bundle.js node viewer/sfm/test_geom.js python3 viewer/sfm/make_nav_ref.py brandenburg && node viewer/sfm/test_nav.js brandenburg `viewer/sfm/make_pixels.py` and `viewer/sfm/test_pipeline.js` run the whole browser pipeline, from pixels to cameras, against the Python on the same images. `viewer/sfm/check_page.py` opens the upload page in headless Chrome and drives it end to end. ## What it does not do - The browser page does not estimate lens distortion, and it takes each photograph's focal length from a 50 degree field of view. The Python pipeline estimates both. - Matching compares every pair of photographs. That is fine up to a few hundred; there is no retrieval step for thousands. - The paper's triangulated morphs, its non-photorealistic rendering, annotation, and registration to an overhead map are not implemented. - There is no dense reconstruction: the cloud is the sparse set of points that matching found. ## Licence The code and the documentation are under the MIT licence, in `LICENSE`. The photographs in `viewer/scenes/brandenburg/images/` are not. Each keeps its own licence, recorded with its author and source in `viewer/scenes/brandenburg/credits.json`, and the files derived from them in that folder (the depth maps, the point colours, the atlas) follow the same terms. Photographs you reconstruct yourself are yours. ## Credits The method is from Noah Snavely, Steven M. Seitz and Richard Szeliski, "Photo Tourism: Exploring Photo Collections in 3D", ACM Transactions on Graphics 25(3), SIGGRAPH 2006. The 79 photographs of the Brandenburg Gate in `viewer/scenes/brandenburg/` are from Wikimedia Commons. Each is under its own licence (Creative Commons or public domain) and carries its author, title, licence and source URL in `viewer/scenes/brandenburg/credits.json`; the explorer shows the credit for each photograph when you visit it. The depth maps in that scene are derived from those photographs. The optional learned matcher uses SuperPoint (DeTone, Malisiewicz and Rabinovich, 2018) and LightGlue (Lindenberger, Sarlin and Pollefeys, 2023) through the `lightglue` package.