ZHIWEN
profile photo

Zhiwen Fan

I am a Ph.D. student in Electrical Engineering at The University of Texas at Austin advised by Prof. Zhangyang Wang at VITA group. Previously, I was a senior algorithm engineer at Alibaba Cloud worked with Prof. Ping Tan and Siyu Zhu. I am a winner of Qualcomm Innovation Fellowship 2022.

CV  /  Research Statement  /  Email  /  Google Scholar  

News

  • 03/2023, one paper has been accepted by CVPR 2023, as hightlight presentation.
  • 01/2023, one paper has been accepted by ICLR 2023.
  • 09/2022, two papers have been accepted by NeurIPS 2022.
  • 07/2022, one paper has been accepted by ASP-DAC 2023.
  • 08/2022, we won Qualcomm Innovation Fellowship (North America) 2022 (QIF 2022).
  • 07/2022, we won 3rd place of University Demo Best Demonstration at 59th Design Automation Conference (DAC 2022).
  • 07/2022, three papers have been accepted by ECCV 2022.
  • 05/2022, one paper has been accepted by ICML 2022.
  • 03/2022, two papers have been accepted by CVPR 2022, including one oral presentation.
  • 10/2021, one paper has been accepted by 3DV 2021.
  • 08/2021, the first version of floorplan CAD dataset is published at here.
  • 07/2021, one paper has been accepted by ICCV 2021.
  • 02/2020, one paper has been accepted by CVPR 2020, and is selected as oral presentation
  • Recent Projects

  • Implicit Neural Representation
  •     Fig1. INS(INR Stylization)           Fig2. NeRF-SOS(RGB).           Fig3. NeRF-SOS(Seg).           Fig4. Single-View NeRF

    INR editing: In our INS paper, we conduct a pilot study for training stylized implicit representations (e.g., SIREN, NeRF, SDF). We obtain faithful stylizations and can interpolate between different styles to generate new mixed style. In our INR-DSP paper, we propose a theoretically grounded signal processing framework for Implicit Neural Representations (INR), which analytically manipulates INRs on the weight space through differential operators. In our NeRF-SOS paper, We propose a novel collaborative contrastive loss for NeRF to segment objects in complex real-world scenes, without any annotation.

    Sparse view NeRF: In our SinNeRF paper, we propose thoughtfully designed semantic and geometry regularizations to train neural radiance field using only a single view.

    NeRF with augmentations: In our Aug-NeRF paper, we propose to augment NeRF with worst-case perturbations in three distinct levels with physical grounds.

  • Efficient MVS and MTL
  • Efficient MVS: We propose Cas-MVSNet and Cas-StereoNet, by formulating cost volume in a coarse to fine manner. We obtain a 23.1% improvement on DTU dataset with 50.6% and 74.2% reduction in GPU memory and run-time. It is also rank 1st within all learning-based methods on Tanks and Temples benchmark. See CVPR2020 for more details.
    Efficient MTL: In our M^3$-ViT paper, we propose to activate any task of interest, by integrating mixture-of-experts (MoE) layers into a ViT backbone, along with hardware-level innovations. M^3-ViT reduce the memory by 2.4x, saving 9.23x energy, on PASCAL-Context and NYUD-v2 datasets.

  • CAD(computer-aided design) Drawing Symbol Spotting
  • Project Page and Product Page

    CAD symbol spotting can be use in architecture, engineering and construction (AEC) industries to accelerate the efficiency of 3D modeling from CAD drawings.
    We release the first large-scale real-world dataset of over 10,000 CAD drawings with line-grained annotations (35 classes), covering various types of builds. We introduce the new task of Panoptic Symbol Spotting, which is a relaxation of the traditional symbol spotting problem. It spots and parse both countable object instances (windows, doors, tables, etc.) and uncountable stuff (wall, railing, etc.) from CAD drawings, Moreover, we propose the Panoptic Quality (PQ) as the evaluation criteria of panotic symbol spotting results.
    PanCADNet: We present a CNN-GCN method in our ICCV2021 which unified a GCN head and a detection head for semantic and instance symbol spotting respectively.
    CADTransformer: We present a transformer-based framework named CADTransormer (CVPR2022), by painlessly modifying existing vision transformer (ViT) backbones.

  • Compressed Sensing MRI and Image Deraining
  • I have also worked on low-level computer vision tasks (e.g. Compressed Sensing MRI and Single Image Deraining) using Deep Neural Network before the year 2019. See IPMI2019, ACM MM2019, ECCV 2018, AAAI 2018 and TIP 2019, MRI 2019, MRI 2019 for details.

    Publications

    I'm interested in devleoping Implicit Neural Representations, Efficient 3D Reconstruction, Transformer Models and Low-level Computer Vision.

    Conference Papers:

    1. StegaNeRF: Embedding Invisible Information within Neural Radiance Fields
      Chenxin Li*, Brandon Y*. Feng, Zhiwen Fan*, Panwang Pan, Zhangyang Wang.
      Preprint | paper | project page| code

    2. NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views
      Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, Zhangyang Wang.
      CVPR 2023 (Highlight) | paper | project page| code

    3. NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes
      Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, Zhangyang Wang.
      ICLR 2023 | paper | project page| code

    4. M^3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
      Zhiwen Fan*, Hanxue Liang*, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang,
      NeurIPS 2022| paper | code

    5. Signal Processing for Implicit Neural Representations
      Dejia Xu*, Peihao Wang*, Yifan Jiang, Zhiwen Fan, Zhangyang Wang.
      NeurIPS 2022| paper | code

    6. Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices
      Zhiwen Fan*, Yimeng Zhang*, Akshay Karkal Kamath*, Qiucheng Wu, Wuyang Chen, Zhangyang Wang, Shiyu Chang, Sijia Liu, Cong Hao,
      ASP-DAC 2023| paper | code

    7. Unified Implicit Neural Stylization
      Zhiwen Fan*, Yifan Jiang*, Peihao Wang*, Xinyu Gong, Dejia Xu, Zhangyang Wang.
      ECCV 2022| paper | project page| code

    8. SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
      Dejia Xu*, Yifan Jiang*, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang.
      ECCV 2022| paper | project page| code

    9. Point Cloud Domain Adaptation via Masked Local 3D Structure Prediction
      Hanxue Liang, Hehe Fan, Zhiwen Fan, Yi Wang, Tianlong Chen, Yu Cheng, Zhangyang Wang.
      ECCV 2022| paper | code

    10. Neural Implicit Dictionary Learning via Mixture-of-Expert Training
      Peihao Wang, Zhiwen Fan, Tianlong Chen, Zhangyang Wang.
      ICML 2022| paper | code

    11. CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings
      Zhiwen Fan, Tianlong Chen, Peihao Wang, Zhangyang Wang
      CVPR 2022 (Oral Presentation) | paper | code

    12. Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Augmentations
      Tianlong Chen*, Peihao Wang*, Zhiwen Fan, Zhangyang Wang
      CVPR 2022 | paper | code

    13. MeshMVS: Multi-View Stereo Guided Mesh Reconstruction
      Rakesh Shrestha, Zhiwen Fan, Qingkun Su, Zuozhuo Dai, Siyu Zhu, Ping Tan
      3DV 2021 | paper | code

    14. FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting
      Zhiwen Fan*,Lingjie Zhu*, Honghua Li, Xiaohao Chen, Siyu Zhu, Ping Tan
      ICCV 2021 | paper | poster | video

    15. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
      Zhiwen Fan*, Xiaodong Gu*, Siyu Zhu, Zuozhuo Dai, Feitong Tan, Ping Tan
      CVPR 2020 (Oral Presentation) | paper | code | video

    16. Joint CS-MRI reconstruction and segmentation with a unified deep network
      Zhiwen Fan*,Liyan Sun*, Xinghao Ding, Yue Huang, John Paisley
      IPMI 2019 | paper

    17. Residual-guide network for single image deraining
      Zhiwen Fan*, Huafeng Wu*, Xueyang Fu, Yue Huang, Xinghao Ding
      ACM MM 2019 | paper

    18. A Segmentation-aware Deep Fusion Network for Compressed Sensing MRI
      Zhiwen Fan*, Liyan Sun*, Xinghao Ding, Yue Huang, Congbo Cai, John Paisley
      ECCV 2018 | paper

    19. Compressed Sensing MRI Using a Recursive Dilated Network
      Zhiwen Fan*, Liyan Sun*, Yue Huang, Xinghao Ding, John Paisley
      AAAI 2018 (* equal contribution) | paper

    20. Two-step approach for single underwater image enhancement
      Xueyang Fu, Zhiwen Fan, Mei Ling, Yue Huang, Xinghao Ding
      ISPACS 2017 | paper

    Journal Papers:

    1. A deep information sharing network for multi-contrast compressed sensing MRI reconstruction
      Liyan Sun, Zhiwen Fan, Xueyang Fu, Yue Huang, Xinghao Ding, John Paisley
      IEEE TIP 2019 |paper

    2. Region-of-interest undersampled MRI reconstruction: A deep convolutional neural network approach
      Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, John Paisley
      Magnetic resonance imaging 2019 | paper

    3. A divide-and-conquer approach to compressed sensing MRI
      Liyan Sun, Zhiwen Fan, Xinghao Ding, Congbo Cai, Yue Huang, John Paisley
      Magnetic resonance imaging 2019 | paper

    Experience

    GOOGLE

    Google, San Francisco, US

    Research Intern
    05/2022--present

    aliabab

    Alibaba Cloud, Hangzhou, China

    Senior Algorithm Engineer
    07/2019--08/2021

    Services

  • Journal Reviewers of TPAMI, TIP, IJCV, Neurocomputing

  • Conference Reviewers of NeurIPS 2022, ICML 2022, CVPR 2022, ICCV 2021, AAAI 2021, ICME 2019

  • Invited Talk

  • [June 2022] "Unified Implicit Neural Stylization" at Xiamen University and Kungfu.ai. Slides

  • Awards

  • 2022, winner of Qualcomm Innovation Fellowship (North America) 2022

  • 2022, 3rd place of University Demo Best Demonstration at 59th Design Automation Conference

  • 2022, Professional Development Award of UT Austin

  • 2019, Outstanding Graduates of Xiamen University

  • 2017 The First Prize Scholarship of Xiamen University

  • 2016 The First Prize Scholarship of Xiamen University

  • 2016, Outstanding Graduates of Shandong Provience