ZHIWEN
profile photo

Zhiwen Fan

I am a Ph.D. student in Electrical Engineering at The University of Texas at Austin advised by Prof. Zhangyang Wang at VITA group. Previously, I was a senior algorithm engineer at Alibaba Cloud worked with Prof. Ping Tan and Siyu Zhu. I am a winner of Qualcomm Innovation Fellowship 2022.

CV  /  Email  /  Google Scholar  

News

  • 09/2022, two papers have been accepted by NeurIPS 2022.
  • 08/2022, we won Qualcomm Innovation Fellowship (North America) 2022 (QIF 2022).
  • 07/2022, we won 3rd place of University Demo Best Demonstration at 59th Design Automation Conference (DAC 2022).
  • 07/2022, three papers have been accepted by ECCV 2022.
  • 05/2022, one paper has been accepted by ICML 2022.
  • 03/2022, two papers have been accepted by CVPR 2022, including one oral presentation.
  • 10/2021, one paper has been accepted by 3DV 2021.
  • 08/2021, the first version of floorplan CAD dataset is published at here.
  • 07/2021, one paper has been accepted by ICCV 2021.
  • 02/2020, one paper has been accepted by CVPR 2020, and is selected as oral presentation
  • Recent Projects

  • Implicit Neural Representation
  • Fig1. Single-View Novel View Synthesis         Fig2. NeRF Stylization                                 Fig3. NeRF Stylization

    INR editing: In our INS paper, we conduct a pilot study for training stylized implicit representations (e.g., SIREN, NeRF, SDF). We obtain faithful stylizations and can interpolate between different styles to generate new mixed style. In our INR-DSP paper, we propose a theoretically grounded signal processing framework for Implicit Neural Representations (INR), which analytically manipulates INRs on the weight space through differential operators.

    Sparse view NeRF: In our SinNeRF paper, we propose thoughtfully designed semantic and geometry regularizations to train neural radiance field using only a single view.

    NeRF augmentations: In our Aug-NeRF paper, we propose to augment NeRF with worst-case perturbations in three distinct levels with physical grounds. They effectively boost NeRF in both novel view synthesis (up to 1.5dB PSNR gain) and underlying geometry reconstruction.

  • Efficient MVS and MTL
  • Efficient MVS: We propose Cas-MVSNet and Cas-StereoNet, by formulating cost volume in a coarse to fine manner. We obtain a 23.1% improvement on DTU dataset with 50.6% and 74.2% reduction in GPU memory and run-time. It is also rank 1st within all learning-based methods on Tanks and Temples benchmark. See CVPR2020 for more details.
    Efficient MTL: In our M^3$-ViT paper, we propose to activate any task of interest, by integrating mixture-of-experts (MoE) layers into a ViT backbone, along with hardware-level innovations. M^3-ViT reduce the memory by 2.4x, saving 9.23x energy, on PASCAL-Context and NYUD-v2 datasets.

  • CAD(computer-aided design) Drawing Symbol Spotting
  • Project Page and Product Page

    CAD symbol spotting can be use in architecture, engineering and construction (AEC) industries to accelerate the efficiency of 3D modeling from CAD drawings.
    We release the first large-scale real-world dataset of over 10,000 CAD drawings with line-grained annotations (35 classes), covering various types of builds. We introduce the new task of Panoptic Symbol Spotting, which is a relaxation of the traditional symbol spotting problem. It spots and parse both countable object instances (windows, doors, tables, etc.) and uncountable stuff (wall, railing, etc.) from CAD drawings, Moreover, we propose the Panoptic Quality (PQ) as the evaluation criteria of panotic symbol spotting results.
    PanCADNet: We present a CNN-GCN method in our ICCV2021 which unified a GCN head and a detection head for semantic and instance symbol spotting respectively.
    CADTransformer: We present a transformer-based framework named CADTransormer (CVPR2022), by painlessly modifying existing vision transformer (ViT) backbones.

  • Compressed Sensing MRI and Image Deraining
  • I have also worked on low-level computer vision tasks (e.g. Compressed Sensing MRI and Single Image Deraining) using Deep Neural Network before the year 2019. See IPMI2019, ACM MM2019, ECCV 2018, AAAI 2018 and TIP 2019, MRI 2019, MRI 2019 for details.

    Publications

    I'm interested in devleoping Implicit Neural Representations, Efficient 3D Reconstruction, Transformer Models and Low-level Computer Vision.

    Conference Papers:

    1. M^3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
      Zhiwen Fan*, Hanxue Liang*, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang,
      NeurIPS 2022| paper | code

    2. Signal Processing for Implicit Neural Representations
      Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang.
      NeurIPS 2022| paper | code

    3. Unified Implicit Neural Stylization
      Zhiwen Fan*, Yifan Jiang*, Peihao Wang*, Xinyu Gong, Dejia Xu, Zhangyang Wang.
      ECCV 2022| paper | project page| code

    4. SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
      Dejia Xu*, Yifan Jiang*, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang.
      ECCV 2022| paper | project page| code

    5. Point Cloud Domain Adaptation via Masked Local 3D Structure Prediction
      Hanxue Liang, Hehe Fan, Zhiwen Fan, Yi Wang, Tianlong Chen, Yu Cheng, Zhangyang Wang.
      ECCV 2022| paper | code

    6. Neural Implicit Dictionary Learning via Mixture-of-Expert Training
      Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang.
      ICML 2022| paper | code

    7. CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
      Zhiwen Fan, Tianlong Chen, Peihao Wang, Zhangyang Wang
      CVPR 2022 (Oral Presentation) | paper | code

    8. Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Augmentations
      Tianlong Chen*, Peihao Wang*, Zhiwen Fan, Zhangyang Wang
      CVPR 2022 | paper | code

    9. MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
      Rakesh Shrestha, Zhiwen Fan, Qingkun Su, Zuozhuo Dai, Siyu Zhu, Ping Tan
      3DV 2021 | paper | code

    10. FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting
      Zhiwen Fan*,Lingjie Zhu*, Honghua Li, Xiaohao Chen, Siyu Zhu, Ping Tan
      ICCV 2021 | paper | poster | video

    11. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
      Zhiwen Fan*, Xiaodong Gu*, Siyu Zhu, Zuozhuo Dai, Feitong Tan, Ping Tan
      CVPR 2020 (Oral Presentation) | paper | code | video

    12. Joint CS-MRI reconstruction and segmentation with a unified deep network
      Zhiwen Fan*,Liyan Sun*, Xinghao Ding, Yue Huang, John Paisley
      IPMI 2019 | paper

    13. Residual-guide network for single image deraining
      Zhiwen Fan*, Huafeng Wu*, Xueyang Fu, Yue Huang, Xinghao Ding
      ACM MM 2019 | paper

    14. A Segmentation-aware Deep Fusion Network for Compressed Sensing MRI
      Zhiwen Fan*, Liyan Sun*, Xinghao Ding, Yue Huang, Congbo Cai, John Paisley
      ECCV 2018 | paper

    15. Compressed Sensing MRI Using a Recursive Dilated Network
      Zhiwen Fan*, Liyan Sun*, Yue Huang, Xinghao Ding, John Paisley
      AAAI 2018 (* equal contribution) | paper

    16. Two-step approach for single underwater image enhancement
      Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, Xinghao Ding
      ISPACS 2017 | paper

    Journal Papers:

    1. A deep information sharing network for multi-contrast compressed sensing MRI reconstruction
      Liyan Sun, Zhiwen Fan, Xueyang Fu, Yue Huang, Xinghao Ding, John Paisley
      IEEE TIP 2019 |paper

    2. Region-of-interest undersampled MRI reconstruction: A deep convolutional neural network approach
      Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, John Paisley
      Magnetic resonance imaging 2019 | paper

    3. A divide-and-conquer approach to compressed sensing MRI
      Liyan Sun, Zhiwen Fan, Xinghao Ding, Congbo Cai, Yue Huang, John Paisley
      Magnetic resonance imaging 2019 | paper

    Experience

    GOOGLE

    Google, San Francisco, US

    Research Intern
    05/2022--present

    aliabab

    Alibaba Cloud, Hangzhou, China

    Senior Algorithm Engineer
    07/2019--08/2021

    Services

  • Journal Reviewers of TPAMI, TIP, IJCV, Neurocomputing

  • Conference Reviewers of NeurIPS 2022, ICML 2022, CVPR 2022, ICCV 2021, AAAI 2021, ICME 2019

  • Invited Talk

  • [June 2022] "Unified Implicit Neural Stylization" at Xiamen University and Kungfu.ai. Slides

  • Awards

  • 2022, winner of Qualcomm Innovation Fellowship (North America) 2022

  • 2022, 3rd place of University Demo Best Demonstration at 59th Design Automation Conference

  • 2022, Professional Development Award of UT Austin

  • 2019, Outstanding Graduates of Xiamen University

  • 2017 The First Prize Scholarship of Xiamen University

  • 2016 The First Prize Scholarship of Xiamen University

  • 2016, Outstanding Graduates of Shandong Provience