可微渲染 SoftRas 实践

GoCoding

发布于 2021-06-22 19:32:08

8390

发布于 2021-06-22 19:32:08

文章被收录于专栏：GoCodingGoCoding

SoftRas[1] 是目前主流三角网格可微渲染器之一。

可微渲染通过计算渲染过程的导数，使得从单张图片学习三维结构逐渐成为现实。可微渲染目前被广泛地应用于三维重建，特别是人体重建、人脸重建和三维属性估计等应用中。

安装

conda 安装 PyTorch 环境：

conda create -n torch python=3.8 -y
conda activate torch

conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -y

conda activate torch
python - <<-EOF
import platform
import torch
print(f"Python : {platform.python_version()}")
print(f"PyTorch: {torch.__version__}")
print(f"  CUDA : {torch.version.cuda}")
EOF

Python : 3.8.10
PyTorch: 1.9.0
  CUDA : 11.1

获取代码并安装：

git clone https://github.com/ShichenLiu/SoftRas.git
cd SoftRas
python setup.py install

可设 setup.py 镜像源：

cat <<-EOF > ~/.pydistutils.cfg
[easy_install]
index_url = http://mirrors.aliyun.com/pypi/simple
EOF

应用

安装模型查看工具：

snap install ogre-meshviewer
# 或
snap install meshlab

渲染物体

渲染测试：

CUDA_VISIBLE_DEVICES=0 python examples/demo_render.py

渲染结果：

对比前后模型:

ogre-meshviewer data/obj/spot/spot_triangulated.obj

ogre-meshviewer data/results/output_render/saved_spot.obj

Mesh 重建

下载数据集：

bash examples/recon/download_dataset.sh

训练模型：

$ CUDA_VISIBLE_DEVICES=0 python examples/recon/train.py -eid recon
Loading dataset: 100%|██████████████████████████| 13/13 [00:35<00:00,  2.74s/it]
Iter: [0/250000] Time 1.189 Loss 0.655 lr 0.000100 sv 0.000100
Iter: [100/250000] Time 0.464 Loss 0.405 lr 0.000100 sv 0.000100
...
Iter: [250000/250000] Time 0.450 Loss 0.128 lr 0.000030 sv 0.000030

测试模型：

$ CUDA_VISIBLE_DEVICES=0 python examples/recon/test.py -eid recon \
    -d 'data/results/models/recon/checkpoint_0250000.pth.tar'
Loading dataset: 100%|██████████████████████████| 13/13 [00:03<00:00,  3.25it/s]
Iter: [0/97]    Time 0.419      IoU 0.697
=================================
Mean IoU: 65.586 for class Airplane

Iter: [0/43]    Time 0.095      IoU 0.587
=================================
Mean IoU: 49.798 for class Bench

Iter: [0/37]    Time 0.089      IoU 0.621
=================================
Mean IoU: 68.975 for class Cabinet

Iter: [0/179]   Time 0.088      IoU 0.741
Iter: [100/179] Time 0.083      IoU 0.772
=================================
Mean IoU: 74.224 for class Car

Iter: [0/162]   Time 0.086      IoU 0.565
Iter: [100/162] Time 0.085      IoU 0.522
=================================
Mean IoU: 52.933 for class Chair

Iter: [0/26]    Time 0.094      IoU 0.681
=================================
Mean IoU: 60.553 for class Display

Iter: [0/55]    Time 0.087      IoU 0.526
=================================
Mean IoU: 45.751 for class Lamp

Iter: [0/38]    Time 0.086      IoU 0.580
=================================
Mean IoU: 65.626 for class Loudspeaker

Iter: [0/56]    Time 0.090      IoU 0.783
=================================
Mean IoU: 68.683 for class Rifle

Iter: [0/76]    Time 0.092      IoU 0.647
=================================
Mean IoU: 68.111 for class Sofa

Iter: [0/204]   Time 0.090      IoU 0.405
Iter: [100/204] Time 0.087      IoU 0.435
Iter: [200/204] Time 0.086      IoU 0.567
=================================
Mean IoU: 46.206 for class Table

Iter: [0/25]    Time 0.097      IoU 0.901
=================================
Mean IoU: 82.261 for class Telephone

Iter: [0/46]    Time 0.087      IoU 0.503
=================================
Mean IoU: 61.019 for class Watercraft

=================================
Mean IoU: 62.287 for all classes

Mesh 重建：

# 获取 `softras_recon.py` 进 `examples/recon/`
#   https://github.com/ikuokuo/start-3d-recon/blob/master/samples/softras_recon.py
# 注释 `iou` 直接返回 0，位于 `examples/recon/models.py` `evaluate_iou()`

# 2D 图像重构 3D Mesh
CUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \
    -s '.' \
    -d 'data/results/models/recon/checkpoint_0250000.pth.tar' \
    -img 'data/car_64x64.png'

ogre-meshviewer data/car_64x64.obj

重建图像：

重建结果：

或重建 ShapeNet 数据集内图像：

# mesh recon images of ShapeNet dataset
CUDA_VISIBLE_DEVICES=0 python examples/recon/softras_recon.py \
    -s '.' \
    -d 'data/results/models/recon/checkpoint_0250000.pth.tar' \
    -imgs 'data/datasets/02958343_test_images.npz'

或使用 SoftRas 训练好的模型：

SoftRas trained with silhouettes supervision (62+ IoU): google drive[2]
SoftRas trained with shading supervision (64+ IoU, test with --shading-model arg): google drive[3]
SoftRas reconstructed meshes with color (random sampled): google drive[4]

Jrender[5] - Jittor[6]
- JRender 解读[7]
- 3D Recon[8]

脚注

[1]

SoftRas: https://github.com/ShichenLiu/SoftRas

[2]

google drive: https://drive.google.com/file/d/1GlZJVih5BMGp026mpxK2scWJXqT94VUx/view?usp=sharing

[3]

google drive: https://drive.google.com/file/d/1r63AKNn3ecMho6RFE7gFefRv78Pmbe5h/view?usp=sharing

[4]

google drive: https://drive.google.com/file/d/1gnSshn0k9JpVpoSTWIQoV2QFAlin3AUK/view?usp=sharing

[5]

Jrender: https://github.com/Jittor/jrender

[6]

Jittor: https://cg.cs.tsinghua.edu.cn/jittor/

[7]

JRender 解读: https://zhuanlan.zhihu.com/p/336364959

[8]

3D Recon: https://github.com/Jittor/shapenet-reconstruction-jittor

本文参与腾讯云自媒体分享计划，分享自微信公众号。

原始发表：2021-06-19，如有侵权请联系 cloudcommunity@tencent.com 删除

https

网络安全

github

git

本文分享自 GoCoding 微信公众号，前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

登录后参与评论

0 条评论

热度