Zhiwen Fan

Recent Projects

Implicit Neural Representation

INR editing: In our INS paper, we conduct a pilot study for training stylized implicit representations (e.g., SIREN, NeRF, SDF). We obtain faithful stylizations and can interpolate between different styles to generate new mixed style. In our INR-DSP paper, we propose a theoretically grounded signal processing framework for Implicit Neural Representations (INR), which analytically manipulates INRs on the weight space through differential operators. In our NeRF-SOS paper, We propose a novel collaborative contrastive loss for NeRF to segment objects in complex real-world scenes, without any annotation.

Sparse view NeRF: In our SinNeRF paper, we propose thoughtfully designed semantic and geometry regularizations to train neural radiance field using only a single view.

NeRF with augmentations: In our Aug-NeRF paper, we propose to augment NeRF with worst-case perturbations in three distinct levels with physical grounds.

Efficient MVS and MTL

Efficient MVS: We propose Cas-MVSNet and Cas-StereoNet, by formulating cost volume in a coarse to fine manner. We obtain a 23.1% improvement on DTU dataset with 50.6% and 74.2% reduction in GPU memory and run-time. It is also rank 1st within all learning-based methods on Tanks and Temples benchmark. See CVPR2020 for more details.
Efficient MTL: In our M^3$-ViT paper, we propose to activate any task of interest, by integrating mixture-of-experts (MoE) layers into a ViT backbone, along with hardware-level innovations. M^3-ViT reduce the memory by 2.4x, saving 9.23x energy, on PASCAL-Context and NYUD-v2 datasets.

CAD(computer-aided design) Drawing Symbol Spotting

Project Page and Product Page

CAD symbol spotting can be use in architecture, engineering and construction (AEC) industries to accelerate the efficiency of 3D modeling from CAD drawings.
We release the first large-scale real-world dataset of over 10,000 CAD drawings with line-grained annotations (35 classes), covering various types of builds. We introduce the new task of Panoptic Symbol Spotting, which is a relaxation of the traditional symbol spotting problem. It spots and parse both countable object instances (windows, doors, tables, etc.) and uncountable stuff (wall, railing, etc.) from CAD drawings, Moreover, we propose the Panoptic Quality (PQ) as the evaluation criteria of panotic symbol spotting results.
PanCADNet: We present a CNN-GCN method in our ICCV2021 which unified a GCN head and a detection head for semantic and instance symbol spotting respectively.
CADTransformer: We present a transformer-based framework named CADTransormer (CVPR2022), by painlessly modifying existing vision transformer (ViT) backbones.

Compressed Sensing MRI and Image Deraining

I have also worked on low-level computer vision tasks (e.g. Compressed Sensing MRI and Single Image Deraining) using Deep Neural Network before the year 2019. See IPMI2019, ACM MM2019, ECCV 2018, AAAI 2018 and TIP 2019, MRI 2019, MRI 2019 for details.

Publications

I'm interested in devleoping Implicit Neural Representations, Efficient 3D Reconstruction, Transformer Models and Low-level Computer Vision.

Conference Papers:

1. StegaNeRF: Embedding Invisible Information within Neural Radiance Fields
   Chenxin Li*, Brandon Y*. Feng, Zhiwen Fan*, Panwang Pan, Zhangyang Wang.
   Preprint | paper | project page| code

2. NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views
   Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, Zhangyang Wang.
   CVPR 2023 (Highlight) | paper | project page| code

3. NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes
   Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, Zhangyang Wang.
   ICLR 2023 | paper | project page| code

4. M^3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
   Zhiwen Fan*, Hanxue Liang*, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang,
   NeurIPS 2022| paper | code

5. Signal Processing for Implicit Neural Representations
   Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang.
   NeurIPS 2022| paper | code

6. Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices
   Zhiwen Fan*, Yimeng Zhang*, Akshay Karkal Kamath*, Qiucheng Wu, Wuyang Chen, Zhangyang Wang, Shiyu Chang, Sijia Liu, Cong Hao,
   ASP-DAC 2023| paper | code

7. Unified Implicit Neural Stylization
   Zhiwen Fan*, Yifan Jiang*, Peihao Wang*, Xinyu Gong, Dejia Xu, Zhangyang Wang.
   ECCV 2022| paper | project page| code

8. SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
   Dejia Xu*, Yifan Jiang*, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang.
   ECCV 2022| paper | project page| code

9. Point Cloud Domain Adaptation via Masked Local 3D Structure Prediction
   Hanxue Liang, Hehe Fan, Zhiwen Fan, Yi Wang, Tianlong Chen, Yu Cheng, Zhangyang Wang.
   ECCV 2022| paper | code

10. Neural Implicit Dictionary Learning via Mixture-of-Expert Training
    Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang.
    ICML 2022| paper | code

11. CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
    Zhiwen Fan, Tianlong Chen, Peihao Wang, Zhangyang Wang
    CVPR 2022 (Oral Presentation) | paper | code

12. Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Augmentations
    Tianlong Chen*, Peihao Wang*, Zhiwen Fan, Zhangyang Wang
    CVPR 2022 | paper | code

13. MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
    Rakesh Shrestha, Zhiwen Fan, Qingkun Su, Zuozhuo Dai, Siyu Zhu, Ping Tan
    3DV 2021 | paper | code

14. FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting
    Zhiwen Fan*,Lingjie Zhu*, Honghua Li, Xiaohao Chen, Siyu Zhu, Ping Tan
    ICCV 2021 | paper | poster | video

15. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
    Zhiwen Fan*, Xiaodong Gu*, Siyu Zhu, Zuozhuo Dai, Feitong Tan, Ping Tan
    CVPR 2020 (Oral Presentation) | paper | code | video

16. Joint CS-MRI reconstruction and segmentation with a unified deep network
    Zhiwen Fan*,Liyan Sun*, Xinghao Ding, Yue Huang, John Paisley
    IPMI 2019 | paper

17. Residual-guide network for single image deraining
    Zhiwen Fan*, Huafeng Wu*, Xueyang Fu, Yue Huang, Xinghao Ding
    ACM MM 2019 | paper

18. A Segmentation-aware Deep Fusion Network for Compressed Sensing MRI
    Zhiwen Fan*, Liyan Sun*, Xinghao Ding, Yue Huang, Congbo Cai, John Paisley
    ECCV 2018 | paper

19. Compressed Sensing MRI Using a Recursive Dilated Network
    Zhiwen Fan*, Liyan Sun*, Yue Huang, Xinghao Ding, John Paisley
    AAAI 2018 (* equal contribution) | paper

20. Two-step approach for single underwater image enhancement
    Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, Xinghao Ding
    ISPACS 2017 | paper

Journal Papers:

1. A deep information sharing network for multi-contrast compressed sensing MRI reconstruction
   Liyan Sun, Zhiwen Fan, Xueyang Fu, Yue Huang, Xinghao Ding, John Paisley
   IEEE TIP 2019 |paper

2. Region-of-interest undersampled MRI reconstruction: A deep convolutional neural network approach
   Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, John Paisley
   Magnetic resonance imaging 2019 | paper

3. A divide-and-conquer approach to compressed sensing MRI
   Liyan Sun, Zhiwen Fan, Xinghao Ding, Congbo Cai, Yue Huang, John Paisley
   Magnetic resonance imaging 2019 | paper

Experience