Performance#
Accuracy Performance#
The accuracy performance is measured by calculating the dice score on ground truth and inference results on 7 images. The following table shows the dice score for each class and the mean dice score for each case.
Case |
Image |
Class |
Dice Score |
Mean Dice |
|---|---|---|---|---|
1 |
spleen |
0.9652 |
0.9652 |
|
2 |
aorta |
0.9682 |
0.9682 |
|
3 |
liver |
0.9177 |
0.8008 |
|
hepatic tumor |
0.6840 |
|||
4 |
lung tumor |
0.8779 |
0.8779 |
|
5 |
colon cancer primaries |
0.8421 |
0.8421 |
|
6 |
stomach |
0.9360 |
0.9004 |
|
inferior vena cava |
0.9087 |
|||
pancreas |
0.7810 |
|||
vertebrae L1 |
0.9793 |
|||
vertebrae T8 |
0.9769 |
|||
brain |
0.8203 |
|||
7 |
left rib 8 |
0.8731 |
0.9165 |
|
right rib 3 |
0.9463 |
|||
right rib 12 |
0.9498 |
|||
right iliopsoas |
0.8696 |
|||
heart |
0.9439 |
Speed Performance#
The speed performance is measured by calculating the time cost on 6 images with different GPUs. The following tables show the results for both PyTorch and TensorRT models for each case.
Case 1#
Image: 256cubic.nii.gz
Classes: []
Size: 256 x 256 x 256
GPU |
Model |
Mean Speed (s) |
Median Speed (s) |
|---|---|---|---|
H100 |
PyTorch |
3.028 |
2.968 |
TensorRT |
2.702 |
2.710 |
|
A100 |
PyTorch |
7.126 |
7.147 |
TensorRT |
5.856 |
5.674 |
|
RTX 6000 Ada |
PyTorch |
6.757 |
6.763 |
TensorRT |
6.539 |
6.237 |
|
L40s |
PyTorch |
8.767 |
8.823 |
TensorRT |
8.543 |
8.720 |
Case 2#
Image: 256cubic.nii.gz
Classes: [spleen]
Size: 256 x 256 x 256
GPU |
Model |
Mean Speed (s) |
Median Speed (s) |
|---|---|---|---|
H100 |
PyTorch |
2.511 |
2.521 |
TensorRT |
2.287 |
2.298 |
|
A100 |
PyTorch |
6.117 |
6.125 |
TensorRT |
4.0935 |
4.086 |
|
RTX 6000 Ada |
PyTorch |
5.514 |
5.593 |
TensorRT |
5.169 |
5.168 |
|
L40s |
PyTorch |
6.903 |
6.944 |
TensorRT |
6.597 |
6.594 |
Case 3#
Image: 512cubic.nii.gz
Classes: []
Size: 512 x 512 x 512
GPU |
Model |
Mean Speed (s) |
Median Speed (s) |
|---|---|---|---|
H100 |
PyTorch |
16.308 |
16.294 |
TensorRT |
15.235 |
15.226 |
|
A100 |
PyTorch |
91.567 |
92.848 |
TensorRT |
87.108 |
84.881 |
|
RTX 6000 Ada |
PyTorch |
109.338 |
109.405 |
TensorRT |
103.284 |
103.386 |
|
L40s |
PyTorch |
96.068 |
97.135 |
TensorRT |
95.064 |
91.982 |
Case 4#
Image: 512cubic.nii.gz
Classes: [liver]
Size: 512 x 512 x 512
GPU |
Model |
Mean Speed (s) |
Median Speed (s) |
|---|---|---|---|
H100 |
PyTorch |
14.156 |
14.147 |
TensorRT |
13.079 |
13.066 |
|
A100 |
PyTorch |
37.947 |
37.963 |
TensorRT |
34.156 |
33.852 |
|
RTX 6000 Ada |
PyTorch |
32.242 |
32.256 |
TensorRT |
31.434 |
31.149 |
|
L40s |
PyTorch |
33.164 |
34.715 |
TensorRT |
33.244 |
33.151 |
Case 5#
Image: 512-768.nii.gz
Classes: []
Size: 512 x 512 x 768
GPU |
Model |
Mean Speed (s) |
Median Speed (s) |
|---|---|---|---|
H100 |
PyTorch |
89.312 |
88.864 |
TensorRT |
87.692 |
87.501 |
|
A100 |
PyTorch |
144.568 |
143.292 |
TensorRT |
140.919 |
140.490 |
|
RTX 6000 Ada |
PyTorch |
160.502 |
160.697 |
TensorRT |
151.359 |
144.844 |
|
L40s |
PyTorch |
146.684 |
150.386 |
TensorRT |
147.603 |
144.717 |
Case 6#
Image: 512-768.nii.gz
Classes: [heart]
Size: 512 x 512 x 768
GPU |
Model |
Mean Speed (s) |
Median Speed (s) |
|---|---|---|---|
H100 |
PyTorch |
22.957 |
22.991 |
TensorRT |
21.499 |
21.515 |
|
A100 |
PyTorch |
58.676 |
58.831 |
TensorRT |
55.636 |
54.910 |
|
RTX 6000 Ada |
PyTorch |
49.822 |
50.126 |
TensorRT |
33.956 |
33.831 |
|
L40s |
PyTorch |
49.225 |
51.239 |
TensorRT |
48.805 |
49.690 |