Zhiwen Fan

Recent Projects

Neural Radiance Field

Synthesizing photo-realistic images has been one of the most essential goals in the area of computer vision.

NeRF Augmentations: NeRF often yields inconsistent and visually non-smooth geometric results due to the generalization gap between seen and unseen views. We propose to blend triple level worest-case perturbations to the NeRF training pipeline with physical grouds. In our Aug-NeRF paper, we effectively boosts NeRF accuracy in both novel view synthesis (up to 1.5dB PSNR gain) and underlying geometry reconstruction.

Single view NeRF: NeRF is impeded by the stringent requirement of the dense views captured from multiple well-calibrated cameras, whereas it could be challenging or even infeasible to collect a sufficiently dense coverage of a scene. In our SinNeRF paper, We push the setting of sparse views to the extreme, by training a neural radiance field on only one view with depth information. By generating pseudo labels according to the available single view, the learned radiance field generate more satisfying synthesized results on novel views.

NeRF Stylizatation: In addtion to improve the synthesized image quality, in our INS paper, we conduct a pilot study for training stylized implicit representations (e.g., SIREN, NeRF, SDF) We propose to decouples the ordinary implicit function into a style implicit module and a content implicit module, in order to separately encode the representations from the style image and input scene. An amalgamation module is then applied to aggregate these information and synthesize the stylized output. Consequently, we can synthesize faithful stylization for SIREN, NeRF and SDF. In addition, we can interpolate between different styles and generating images with new mixed style.

CAD(computer-aided design) Drawing Symbol Spotting

Project Page and Product Page

CAD symbol spotting can be use in architecture, engineering and construction (AEC) industries to accelerate the efficiency of 3D modeling from CAD drawings.
We release the first large-scale real-world dataset of over 15,000 CAD drawings with line-grained annotations (35 classes), covering various types of builds. We introduce the new task of Panoptic Symbol Spotting, which is a relaxation of the traditional symbol spotting problem. It spots and parse both countable object instances (windows, doors, tables, etc.) and uncountable stuff (wall, railing, etc.) from CAD drawings, Moreover, we propose the Panoptic Quality (PQ) as the evaluation criteria of panotic symbol spotting results.
To tackle the new proposed problem, we first present a CNN-GCN method in our ICCV2021 which unified a GCN head and a detection head for semantic and instance symbol spotting respectively. Recently, we present a transformer-based framework named CADTransformer, in our CVPR2022, by painlessly modifying existing vision transformer (ViT) backbones to tackle the panoptic symbol spotting task. The PQ is boosted from 0.595 in the GCN-CNN based methods to a new state-of-the-art 0.685.

Efficient Multi-view Stereo and Stereo Matching

To tackle the high computaitonal cost of the existing cost volume-based deep MVS and stereo matching methods, we propose a memory and run time efficient cost volume formulation which is built upon a standard feature pyramid encoding geometry and context at gradually finer scales. Within our new design, We narrow the depth (or disparity) range of each stage by the depth (or disparity) map from the previous stage to recover the output in a coarser to fine manner.
By applying the cascade cost volume to the representative MVS-Net, and obtain a 23.1% improvement on DTU dataset with 50.6% and 74.2% reduction in GPU memory and run-time. It is also rank 1st within all learning-based methods on Tanks and Temples benchmark. In addition, we adapt GwcNet with our proposed cost volume design, and the accuracy ranking rises from 29 to 17 with 37.0% memory reduction on KITTI 2015 test set. See CVPR2020 for more details.

Compressed Sensing MRI and Image Deraining

I have also worked on low-level computer vision tasks (e.g. Compressed Sensing MRI and Single Image Deraining) using Deep Neural Network before the year 2019. See IPMI2019, ACM MM2019, ECCV 2018, AAAI 2018 and TIP 2019, MRI 2019, MRI 2019 for details.

Research

I'm interested in devleoping Neural Radiance Field, Efficient 3D Models, Graph Neural Networks for vector graphics and 3D data and Low-level Computer Vision.

Arxiv Submissions:

1. Unified Implicit Neural Stylization
   Zhiwen Fan*, Yifan Jiang*, Peihao Wang*, Xinyu Gong, Dejia Xu, Zhangyang Wang.
   Arxiv 2022| paper | project page| code

2. SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
   Dejia Xu*, Yifan Jiang*, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang.
   Arxiv 2022| paper | project page| code

Conference Papers:

1. Neural Implicit Dictionary Learning via Mixture-of-Expert Training
   Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang.
   ICML 2022| paper | code

2. CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
   Zhiwen Fan, Tianlong Chen, Peihao Wang, Zhangyang Wang
   CVPR 2022 (Oral Presentation) | paper | code

3. Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Augmentations
   Tianlong Chen, Peihao Wang, Zhiwen Fan, Zhangyang Wang
   CVPR 2022 | paper | code

4. MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
   Rakesh Shrestha, Zhiwen Fan, Qingkun Su, Zuozhuo Dai, Siyu Zhu, Ping Tan
   3DV 2021 | paper | code

5. FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting
   Zhiwen Fan*, Lingjie Zhu*, Honghua Li, Xiaohao Chen, Siyu Zhu, Ping Tan
   ICCV 2021 | paper | poster | video

6. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
   Xiaodong Gu*, Zhiwen Fan*, Siyu Zhu, Zuozhuo Dai, Feitong Tan, Ping Tan
   CVPR 2020 (Oral Presentation) | paper | code | video

7. Joint CS-MRI reconstruction and segmentation with a unified deep network
   Liyan Sun*, Zhiwen Fan*, Xinghao Ding, Yue Huang, John Paisley
   IPMI 2019 | paper

8. Residual-guide network for single image deraining
   Zhiwen Fan*, Huafeng Wu*, Xueyang Fu, Yue Huang, Xinghao Ding
   ACM MM 2019 | paper

9. A Segmentation-aware Deep Fusion Network for Compressed Sensing MRI
   Zhiwen Fan*, Liyan Sun*, Xinghao Ding, Yue Huang, Congbo Cai, John Paisley
   ECCV 2018 | paper

10. Compressed Sensing MRI Using a Recursive Dilated Network
    Liyan Sun*, Zhiwen Fan*, Yue Huang, Xinghao Ding, John Paisley
    AAAI 2018 (* equal contribution) | paper

11. Two-step approach for single underwater image enhancement
    Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, Xinghao Ding
    ISPACS 2017 | paper

Journal Papers:

1. A deep information sharing network for multi-contrast compressed sensing MRI reconstruction
   Liyan Sun*, Zhiwen Fan*, Xueyang Fu, Yue Huang, Xinghao Ding, John Paisley
   IEEE TIP 2019 |paper

2. Region-of-interest undersampled MRI reconstruction: A deep convolutional neural network approach
   Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, John Paisley
   Magnetic resonance imaging 2019 | paper

3. A divide-and-conquer approach to compressed sensing MRI
   Liyan Sun, Zhiwen Fan, Xinghao Ding, Congbo Cai, Yue Huang, John Paisley
   Magnetic resonance imaging 2019 | paper

Experience