profile photo

Zhiwen Fan

I am a Ph.D. student in Electrical Engineering at The University of Texas at Austin advised by Prof. Zhangyang Wang at VITA group.
Previously, I was senior algorithm engineer at Alibaba Cloud worked with Prof. Ping Tan and Siyu Zhu.

CV  /  Email  /  Google Scholar  


  • 05/2022, one paper have been accepted by ICML 2022.
  • 03/2022, two papers have been accepted by CVPR 2022, including one oral presentation.
  • 10/2021, one paper has been accepted by 3DV 2021.
  • 08/2021, the first version of floorplan CAD dataset is published at here.
  • 07/2021, one paper has been accepted by ICCV 2021.
  • 02/2020, one paper has been accepted by CVPR 2020, and is selected as oral presentation
  • Recent Projects

  • Neural Radiance Field
  • Synthesizing photo-realistic images has been one of the most essential goals in the area of computer vision.

    NeRF Augmentations: NeRF often yields inconsistent and visually non-smooth geometric results due to the generalization gap between seen and unseen views. We propose to blend triple level worest-case perturbations to the NeRF training pipeline with physical grouds. In our Aug-NeRF paper, we effectively boosts NeRF accuracy in both novel view synthesis (up to 1.5dB PSNR gain) and underlying geometry reconstruction.

    Single view NeRF: NeRF is impeded by the stringent requirement of the dense views captured from multiple well-calibrated cameras, whereas it could be challenging or even infeasible to collect a sufficiently dense coverage of a scene. In our SinNeRF paper, We push the setting of sparse views to the extreme, by training a neural radiance field on only one view with depth information. By generating pseudo labels according to the available single view, the learned radiance field generate more satisfying synthesized results on novel views.

    NeRF Stylizatation: In addtion to improve the synthesized image quality, in our INS paper, we conduct a pilot study for training stylized implicit representations (e.g., SIREN, NeRF, SDF) We propose to decouples the ordinary implicit function into a style implicit module and a content implicit module, in order to separately encode the representations from the style image and input scene. An amalgamation module is then applied to aggregate these information and synthesize the stylized output. Consequently, we can synthesize faithful stylization for SIREN, NeRF and SDF. In addition, we can interpolate between different styles and generating images with new mixed style.

  • CAD(computer-aided design) Drawing Symbol Spotting
  • Project Page and Product Page

    CAD symbol spotting can be use in architecture, engineering and construction (AEC) industries to accelerate the efficiency of 3D modeling from CAD drawings.
    We release the first large-scale real-world dataset of over 15,000 CAD drawings with line-grained annotations (35 classes), covering various types of builds. We introduce the new task of Panoptic Symbol Spotting, which is a relaxation of the traditional symbol spotting problem. It spots and parse both countable object instances (windows, doors, tables, etc.) and uncountable stuff (wall, railing, etc.) from CAD drawings, Moreover, we propose the Panoptic Quality (PQ) as the evaluation criteria of panotic symbol spotting results.
    To tackle the new proposed problem, we first present a CNN-GCN method in our ICCV2021 which unified a GCN head and a detection head for semantic and instance symbol spotting respectively. Recently, we present a transformer-based framework named CADTransformer, in our CVPR2022, by painlessly modifying existing vision transformer (ViT) backbones to tackle the panoptic symbol spotting task. The PQ is boosted from 0.595 in the GCN-CNN based methods to a new state-of-the-art 0.685.

  • Efficient Multi-view Stereo and Stereo Matching
  • To tackle the high computaitonal cost of the existing cost volume-based deep MVS and stereo matching methods, we propose a memory and run time efficient cost volume formulation which is built upon a standard feature pyramid encoding geometry and context at gradually finer scales. Within our new design, We narrow the depth (or disparity) range of each stage by the depth (or disparity) map from the previous stage to recover the output in a coarser to fine manner.
    By applying the cascade cost volume to the representative MVS-Net, and obtain a 23.1% improvement on DTU dataset with 50.6% and 74.2% reduction in GPU memory and run-time. It is also rank 1st within all learning-based methods on Tanks and Temples benchmark. In addition, we adapt GwcNet with our proposed cost volume design, and the accuracy ranking rises from 29 to 17 with 37.0% memory reduction on KITTI 2015 test set. See CVPR2020 for more details.

  • Compressed Sensing MRI and Image Deraining
  • I have also worked on low-level computer vision tasks (e.g. Compressed Sensing MRI and Single Image Deraining) using Deep Neural Network before the year 2019. See IPMI2019, ACM MM2019, ECCV 2018, AAAI 2018 and TIP 2019, MRI 2019, MRI 2019 for details.


    I'm interested in devleoping Neural Radiance Field, Efficient 3D Models, Graph Neural Networks for vector graphics and 3D data and Low-level Computer Vision.

    Arxiv Submissions:

    1. Unified Implicit Neural Stylization
      Zhiwen Fan*, Yifan Jiang*, Peihao Wang*, Xinyu Gong, Dejia Xu, Zhangyang Wang.
      Arxiv 2022| paper | project page| code

    2. SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
      Dejia Xu*, Yifan Jiang*, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang.
      Arxiv 2022| paper | project page| code

    Conference Papers:

    1. Neural Implicit Dictionary Learning via Mixture-of-Expert Training
      Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang.
      ICML 2022| paper | code

    2. CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
      Zhiwen Fan, Tianlong Chen, Peihao Wang, Zhangyang Wang
      CVPR 2022 (Oral Presentation) | paper | code

    3. Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Augmentations
      Tianlong Chen, Peihao Wang, Zhiwen Fan, Zhangyang Wang
      CVPR 2022 | paper | code

    4. MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
      Rakesh Shrestha, Zhiwen Fan, Qingkun Su, Zuozhuo Dai, Siyu Zhu, Ping Tan
      3DV 2021 | paper | code

    5. FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting
      Zhiwen Fan*, Lingjie Zhu*, Honghua Li, Xiaohao Chen, Siyu Zhu, Ping Tan
      ICCV 2021 | paper | poster | video

    6. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
      Xiaodong Gu*, Zhiwen Fan*, Siyu Zhu, Zuozhuo Dai, Feitong Tan, Ping Tan
      CVPR 2020 (Oral Presentation) | paper | code | video

    7. Joint CS-MRI reconstruction and segmentation with a unified deep network
      Liyan Sun*, Zhiwen Fan*, Xinghao Ding, Yue Huang, John Paisley
      IPMI 2019 | paper

    8. Residual-guide network for single image deraining
      Zhiwen Fan*, Huafeng Wu*, Xueyang Fu, Yue Huang, Xinghao Ding
      ACM MM 2019 | paper

    9. A Segmentation-aware Deep Fusion Network for Compressed Sensing MRI
      Zhiwen Fan*, Liyan Sun*, Xinghao Ding, Yue Huang, Congbo Cai, John Paisley
      ECCV 2018 | paper

    10. Compressed Sensing MRI Using a Recursive Dilated Network
      Liyan Sun*, Zhiwen Fan*, Yue Huang, Xinghao Ding, John Paisley
      AAAI 2018 (* equal contribution) | paper

    11. Two-step approach for single underwater image enhancement
      Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, Xinghao Ding
      ISPACS 2017 | paper

    Journal Papers:

    1. A deep information sharing network for multi-contrast compressed sensing MRI reconstruction
      Liyan Sun*, Zhiwen Fan*, Xueyang Fu, Yue Huang, Xinghao Ding, John Paisley
      IEEE TIP 2019 |paper

    2. Region-of-interest undersampled MRI reconstruction: A deep convolutional neural network approach
      Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, John Paisley
      Magnetic resonance imaging 2019 | paper

    3. A divide-and-conquer approach to compressed sensing MRI
      Liyan Sun, Zhiwen Fan, Xinghao Ding, Congbo Cai, Yue Huang, John Paisley
      Magnetic resonance imaging 2019 | paper



    Google, San Francisco, US

    Research Intern


    Alibaba Cloud, Hangzhou, China

    Senior Algorithm Engineer


  • Journal Reviewers of IJCV, Neurocomputing

  • Conference Reviewers of ICML 2022, CVPR 2022, ICCV 2021, AAAI 2021, ICME 2019

  • Awards

  • 2019, Outstanding Graduates of Xiamen University

  • 2017 The First Prize Scholarship of Xiamen University

  • 2016 The First Prize Scholarship of Xiamen University

  • 2016, Outstanding Graduates of Shandong Provience