Output video improvements - #71
Open
jasper-tms wants to merge 19 commits into
Open
Conversation
These files implement the pictorial-structures pose correction described in Figure 10 of the 2019 DeepFly3D eLife paper. They were deleted in: - 6214d05 (2021-12-19) deepfly/belief_propagation.py - d1798cb (2021-12-20) deepfly/optim_util.py - a073020 (2021-12-20) deepfly/cv_util.py Restored verbatim from the parent of each deletion commit, into df3d/ rather than the old deepfly/ package. Imports still reference `from deepfly.*` and will not run as-is; subsequent commits will adapt: - deepfly.Config -> df3d.config - deepfly.Camera -> pyba.Camera (API differs) - deepfly.cv_util -> df3d.cv_util (kept local) - deepfly.optim_util -> df3d.optim_util - deepfly.logger -> df3d.logger
The pre-refactor implementation called self.get_joint_reprojection_error against self.camNetLeft / self.camNetRight (a separate CameraNetwork per half of the camera ring). Those attributes have not existed since the 2021 deletion of deepfly/Camera.py and deepfly/CameraNetwork.py, so the helper raised AttributeError at the first call from next_error_in_range -- but the GUI buttons that drove that path were commented out, hiding the breakage. Replace with a cached per-camera per-frame per-joint reprojection error norm tensor derived from pyba.CameraNetwork.reprojection_error(reduce= False). Max across cameras is compared to the per-joint threshold in config['reproj_thr'] (default 40 px). Mirrors original behaviour minus the left/right split, which has no analogue in the current single- CameraNetwork pipeline.
The launcher had three bugs that prevented df3d-gui from starting: 1. parse_cli_args double-used argv[2] as both num_images_max (int) and output_subfolder (string). 2. The startup assertion looked for df3d_result*.pkl in input_folder/ output_subfolder, but df3d-cli writes results to <input_folder>_df3d (a sibling, not a subfolder). 3. Core's constructor's second positional argument is output_folder (full path), not output_subfolder; the GUI passed a bare subfolder name like 'df3d' instead, breaking save_path. Switch to argparse, mirror df3d-cli's --output-folder flag and default (input_folder + '_df3d'), and drop the now-unused output-subdir prompt.
These were commented out because Core.joint_has_error was broken (called attributes that no longer exist post-2021 refactor). With that helper repaired in the previous commit, the buttons can be wired back up. The checkbox_solve_bp line is intentionally still disabled -- belief- propagation pose correction itself is not yet plumbed back through Core, so the auto-correct toggle would have nothing to call.
Mechanical adaptations applied to the verbatim-resurrected modules so they can at least be loaded by the current df3d/pyba layout: * Renamed imports: deepfly.Config -> df3d.config, deepfly.cv_util -> df3d.cv_util, deepfly.optim_util -> df3d.optim_util, deepfly.Camera -> pyba.Camera, deepfly.logger -> df3d.logger. * Deferred config['image_shape'] / config['num_peak'] / etc. defaults from kwarg-definition time to function-body time; image_shape is only populated by df3d.core.Core at runtime, so import-time evaluation of those defaults raised KeyError. * Replaced numpy aliases removed in modern numpy: np.int -> int, np.float -> float. * Added a single-point project shim (_project_single) so error_reprojection and project_on_last keep working against pyba.Camera.project's batched (T, J, 3) -> (T, J, 2) signature. * Documented the three remaining API gaps to pyba.Camera at the top of belief_propagation.py: cam.cam_id, cam.get_heatmap, Camera.hm_to_pred. Filling these is part of the heatmap-plumbing work, not this commit.
Adds the data-collection side of the pictorial-structures pose-correction restoration: Core.pose2d_estimation can now optionally fetch the top-K per-channel heatmap peaks (via the new df2d.heatmap_peaks plumbing) and store them in the result pkl under 'top_k_peaks', shape [n_cameras, n_frames, 38, K, 3] with last axis (y, x, score). Peaks are expanded from 19 to 38 joints mirroring points2d, including the right-side camera horizontal-flip (applied only to real peaks; padded slots stay (0, 0, 0) so the score==0 sentinel still distinguishes them). K defaults to config['num_peak'] (=10) -- one of the orphaned BP config parameters that has been sitting unused since 2021. Opt-in only: the flag is off by default, so the existing pkl schema is unchanged for users who do not want belief propagation. Sanity-checked end-to-end against tests/data/reference: peak[0] coordinates match the existing points2d argmax for every cam/frame/joint, and the highest peak score matches heatmap_confidence.
…am.heatmaps belief_propagation.py: Use df2d.util.heatmap_peaks for top-K peak extraction (the pyba.Camera dependency is gone). Added _top_k_peaks_xy_normalized which wraps heatmap_peaks and swaps (y, x) -> (x, y) at the convention boundary (BP uses pyba's (x, y) order internally). Looking up heatmaps now goes through cam.heatmaps[img_id, j_id] directly -- df3d.core.Core will attach the network's heatmaps to each pyba.Camera before invoking BP. Updated module docstring to drop the API-gap warnings; all three blockers (cam_id, get_heatmap, hm_to_pred) are resolved. optim_util.py: Fixed a latent coord-order bug: config['heatmap_shape'] is stored as (H, W) to match the network's row-major axes, but the code multiplied (x, y) points by it -- which would put y in [0, H) and x in [0, W) ranges that disagree with hm[y, x] indexing. Now reverse to (W, H) before multiplication. image_shape is already (W, H) in config so it stays as-is. probability_heatmap now reads cam.heatmaps[img_id, j_id] instead of the deleted cam.get_heatmap method.
Adds Core.run_belief_propagation() and a corresponding --belief-propagation CLI flag. When the flag is set: * pose2d_estimation retains the full per-joint heatmaps from inference (keep_heatmaps=True) -- expanded to 38-joint padded form and with the right-side cameras' heatmaps horizontally flipped to match the saved points2d's un-flipped image coordinates. About 8.7 GB / 1000 frames; not persisted. * After calibrate_calc(), each pyba.Camera in self.camNet is given its heatmaps, BP runs per frame, and corrected (x_norm, y_norm) points are swapped back to df3d's (y_norm, x_norm) convention and written into both self.points2d and each cam.points2d (so the subsequent save() triangulates from corrected detections). * The middle camera is excluded from BP because the current pipeline routes no per-joint heatmaps to it -- including it would empty the itertools.product over per-camera candidate peaks. * Cameras that don't see a given joint (or whose peak list is empty) now contribute a single placeholder peak to keep the product non-empty; placeholders are filtered out via camera_see_joint downstream. Verified end-to-end on tests/data/reference (15 frames): BP changes 20% of the per-cam per-joint slots, mean reprojection error barely moves (1.87 -> 1.86 px) but the worst outlier drops from 43.94 to 16.08 px -- which is exactly the failure mode (a few egregious 2D outliers) that BP is supposed to fix via cross-view geometric consistency. Also: silenced a debug 'not float' print in optim_util.prob_from_heatmap that fired hundreds of times per frame during the BP loop.
The saved points3d was in an arbitrary procrustes-template frame (~94 deg off the fly body), so a coordinate like "tarsus-tip x" mixed anterior- posterior, medial-lateral and dorsal-ventral motion. New df3d.body_align computes a body frame per recording from the six thorax-coxa landmarks and the dorsal stripe, and rotates every point into it: x = anterior-posterior (+x anterior) y = medial-lateral (+y the fly's left) z = dorsal-ventral (+z dorsal / leg lift) Applied in Core.save() and Core.get_points3d(), gated by the config key align_body_axes (default True); points3d is stamped body_axis_aligned so downstream loaders can align legacy results without rotating twice. points3d_wo_procrustes keeps the raw frame. The transform is a proper rigid rotation (det +1, no reflection) about the coxae centroid, so joint angles and all relative geometry are unchanged (verified bit-for-bit against df3dPostProcessing). The ~94 deg offset came from the procrustes template, data/df3d_result.pkl, an old result stored in its own camera frame that every recording is aligned to. That file is now rotated into the body frame too, so procrustes output arrives 0.5 deg from aligned instead of 96 deg, and callers get roughly correct axes even with align_body_axes off. The explicit rotation stays, so the axes are still guaranteed for any other template. Rotating the template only re-orients the output: procrustes runs with scaling=False, so it applies an orthogonal transform whatever it fits, and every within-side inter-joint distance is unchanged to 4e-15. The test_calibration reference moved by the same transform; points3d_wo_procrustes predates procrustes and is untouched. Also stop applying rotate_points3d() in get_points3d(). That transform swaps and negates y and z, which has determinant -1: it is a reflection, not a rotation, and existed only to make the old arbitrary template frame display upright. It mirrored the fly, so +y meant the fly's right in get_points3d() but its left in save(). With the template upright it is simply wrong, so it is gone, and its docstring now warns against reintroducing it. test_body_axis_alignment covers the feature, asserts both code paths agree on the medial-lateral sign, and asserts the shipped template is still body-aligned. test_video_3d remains failing: it already failed on clean master from a matplotlib version mismatch, and its reference is now stale on content too since get_points3d() no longer reflects. Regenerating it needs an environment that reproduces the original rendering.
Add a bone_lines kwarg to plot_drosophila_3d and _compute_3d_img: when given, the existing per-bone Line3D artists are updated via set_data_3d (and the associated figure reused) instead of rebuilding a fresh Figure/Axes3D/lines every frame. make_pose3d_video now builds one canvas per camera via _make_3d_canvas and passes bone_lines into every per-frame _compute_3d_img call. Callers that pass no bone_lines get the original create+close-fig behavior.
Adds a boolean (default in config['only_render_legs']) that, when set, zeroes antenna and stripe joints in each camera's stored 2D points (after bundle adjustment) so pyba skips drawing them, and filters those joints out of the 3D plot via draw_joints.
Previously every 2d thumbnail was force-resized to a fixed 200x100 cell, which stretched non-2:1 input frames. The whole stacked frame was then upsampled to 5000px wide, leaving images visibly pixelated. Each row now has 3 cells of cell_width px (= video_width // 3). 2d thumbnails preserve source aspect; 3d figures rasterize directly at cell_width square. No final cv2 upsample. Module-level constants are in pixels — no more dpi/inches. Default video width is 1920 (1080p). Reference test videos regenerated for the new dimensions.
When camera_{cam_id}.mp4 (or .avi) exists in input_folder, pass that
path to pyba.CameraNetwork instead of the per-frame jpg template.
pyba's Camera.get_image then streams frames via cv2.VideoCapture,
which can be up to ~10x faster than per-frame cv2.imread over a network
mounted server.
Requires the matching pyba change that taught Camera to accept a video
file as image_path.
Also label the 2D and 3D render tqdm bars with a desc + total so
their progress is distinguishable and shows an ETA.
Replace the hardcoded video_width=1920 with output_video_downsampling (default 2) and derive per-cell width from the source frame width at render time. _make_3d_figure / _make_3d_canvas now take cell_width explicitly; _compute_3d_img gets a cell_width keyword arg (default 480) for standalone use of its bone_lines=None fallback.
only_render_legs (now default True) previously hid antenna/stripe joints by zeroing them directly on the camera network before triangulation/save. That corrupted their saved points3d to (0, 0, 0), which in turn broke df3d.body_align's dorsal-ventral sign disambiguation (it reads the stripe's 3D position to tell which way is "up"), flipping the whole 3D skeleton upside down. Replaced the mutating mask with Core.hidden_joints(), a pure query consumed only by plot_2d's new hidden_joints= parameter (paired with a PyBundleAdjustment commit adding that parameter to Camera.plot_2d/plot_reprojections) -- rendering can now hide joints without touching any data calibration, triangulation or alignment depend on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.