Commit 68cf31d9 authored by Anis Kacem's avatar Anis Kacem
Browse files

update evaluation doc

parent 84e2133b
...@@ -19,8 +19,11 @@ Consist of two directed distances: ...@@ -19,8 +19,11 @@ Consist of two directed distances:
These distances are inspired from [1] but have been adpated to fit the problem at hand. These distances are inspired from [1] but have been adpated to fit the problem at hand.
The directed distance $`d_{AB}`$ between meshes $`A`$ and $`B`$ is The directed distance $`d_{AB}`$ between meshes $`A`$ and $`B`$ is
approximated in practice by sampling points on $`A`$ and computing their approximated in practice by sampling points on $`A`$ and computing their distances to the nearest triangles in mesh $`B`$.
distances to the nearest triangles in mesh $`B`$.
<img align="right" height="200" src="data/points12-example.png">
These points are sampled according to the area of the triangles. An example of these points is given in this figure:
The directed distances $`d_{RE}`$ and $`d_{ER}`$ are given by, The directed distances $`d_{RE}`$ and $`d_{ER}`$ are given by,
```math ```math
...@@ -46,7 +49,7 @@ This results in two shape distance values ($`d_{ER}^{shape}`$, $`d_{RE}^{shape}` ...@@ -46,7 +49,7 @@ This results in two shape distance values ($`d_{ER}^{shape}`$, $`d_{RE}^{shape}`
Good estimations are expected to have low shape and texture distance values. Good estimations are expected to have low shape and texture distance values.
### 2. Surface hit-rates ### 2. Surface hit-rates
ith the point-to-triangle distance used above.
Consist of two rates that are computed in two directions: Consist of two rates that are computed in two directions:
1. $`h_{ER}`$ computed from estimation to reference 1. $`h_{ER}`$ computed from estimation to reference
...@@ -61,7 +64,7 @@ Let us consider: ...@@ -61,7 +64,7 @@ Let us consider:
- $`H_{AB}`$: number of points of the source mesh $`A`$ that hit the target $`B`$ - $`H_{AB}`$: number of points of the source mesh $`A`$ that hit the target $`B`$
- $`M_{AB}`$: number of points of the source mesh $`A`$ that miss the target $`B`$. - $`M_{AB}`$: number of points of the source mesh $`A`$ that miss the target $`B`$.
The hit-rate from $`A`$ to $`B`ith the point-to-triangle distance used above.$ is then given by, The hit-rate from $`A`$ to $`B`$ is then given by,
```math ```math
h_{AB} = \frac{H_{AB}}{H_{AB} + M_{AB}} . h_{AB} = \frac{H_{AB}}{H_{AB} + M_{AB}} .
``` ```
...@@ -82,10 +85,10 @@ These areas are then normalized as follows, ...@@ -82,10 +85,10 @@ These areas are then normalized as follows,
The area score $`S_a`$ is then given by, The area score $`S_a`$ is then given by,
```math ```math
S_a = 1 - | \bar{A_{R}} - \bar{A_{E}} | . S_a = 1 - | \bar{A_{R}} - \bar{A_{E}} | = 1 - | \frac{1 - A_{E} / A_{R}}{1 + A_{E} / A_{R}} | .
``` ```
This score results in a value in [0,1]. Good estimations are expected to have high area scores. This score results in a value in [0,1]. Good estimations are expected to have high area scores. It is important to note that such score is not symmetric: estimations with larger triangle areas than references are less penalized than those with smaller triangle areas than references.
### Final score ### Final score
...@@ -108,7 +111,7 @@ The final score is finally given by, ...@@ -108,7 +111,7 @@ The final score is finally given by,
S = \frac{1}{2} S_a(S_s + S_t) . S = \frac{1}{2} S_a(S_s + S_t) .
``` ```
Note that challenge 1 is evaluated on specific regions of the body mesh. The head and hands are ignored, rest of the body is considered. In challenge 2, the full mesh is considered for evaluation. Note that challenge 1 is evaluated on specific regions of the body mesh. The head and hands are ignored, rest of the body is considered. In this case, masks are computed on the reference and included in the computation of the scores. In challenge 2, the full mesh is considered for evaluation.
...@@ -125,11 +128,94 @@ Notation: ...@@ -125,11 +128,94 @@ Notation:
The quality of the estimations, $`Y'`$ and $`E'`$, are evaluated quantitatively with respect to the ground-truth, $`Y`$ and $`E`$, respectively, using a metric evaluating: The quality of the estimations, $`Y'`$ and $`E'`$, are evaluated quantitatively with respect to the ground-truth, $`Y`$ and $`E`$, respectively, using a metric evaluating:
- the shape reconstruction of the corrupted scans `X` with respect to CAD models `Y`. - Shape reconstruction: the shape reconstruction of the corrupted scans `X` with respect to CAD models `Y`.
- the recovery of the sharp edges with respect to ground-truth edges `E`. - Edge recovery: the recovery of the sharp edges with respect to ground-truth edges `E`.
### Shape reconstruction
The shape reconstruction is evaluated on $`Y'`$ with respect to the corresponding CAD model $`Y`$ using a similar formulation to SHARP [challenges 1 and 2](evaluation.md#quantitative-evaluation-of-challenges-12).
The first subscore ['surface-to-surface distances'](evaluation.md#1-surface-to-surface-distances) only involves the shape scores of the meshes ( $`Y'`$, $`Y`$) by computing ($`d_{ER}^{shape}`$, $`d_{RE}^{shape}`$) with the same manner described [here](evaluation.md#1-surface-to-surface-distances). <img align="right" height="100" src="data/points3-example.png">
In order to better reflect the shape reconstrution around the sharp edges on the shape score, additional points are sampled on the triangles nearby the sharp edges. An example of these additional points is shown in this figure:
The hit-rates [hit-rates](evaluation.md#2-surface-hit-rates) and the [surface area scores](evaluation.md#3-surface-area-scores) are also computed following a similar manner as described for [challenges 1 and 2](evaluation.md#quantitative-evaluation-of-challenges-12). The only difference consists of making the surface area scores more symmetric than in challenges 1 and 2 where the focus was on shape completion. In particular, the shape area score is computed as follows:
```math
S_a = 1 - | \frac{1 - \alpha (A_{E} / A_{R} + \beta)}{1 + \alpha (A_{E} / A_{R} + \beta)} | .
```
where $`\alpha`$ and $`\beta`$ are two parameters allowing the control of the symmetry of the score function. The values of these parameters were set to $`\alpha=2.6`$ and $`\beta = -0.6`$.
Finally, the three subscores are combined as described [here](evaluation.md#final-score) to obtain a final shape reconstuction score $`S_s`$.
<img align="right" height="100" src="data/innerparts.png">
Note that the ground-truth CAD models $`Y`$ might contain some inner-parts which are missing in the provided input scans $`X`$. These parts are not visible and cannot be scanned. Consequently, these parts are masked and not included in computing the shape scores. An example of these parts and the masked version of the CAD model is shown in this figure:
### Edge recovery
In addition to the shape reconstruction, the estimated sharp edges $`E'`$ are evaluated with respect to the ground-truth sharp edges $`E`$. The evaluation of the edges is conducted using two criterions:
#### 1. Edge-to-edge distances
Consist of two directed distances:
1. $`d_{ER}^E`$ is computed from the estimation to the reference
2. $`d_{RE}^E`$ is computed from the reference to the estimation.
The directed distance $`d_{AB}^E`$ between edges $`A`$ and $`B`$ is
approximated in practice by sampling points on $`A`$ and computing their distances to the nearest lines in edges $`B`$.
<img align="right" height="100" src="data/pointsedges-example.png">
The points are sampled according to the length of the edges. An example of these points is given in this figure (top: sampled points; bottom: original edge lines):
For each sampled point in $`A`$, the euclidean distance to the nearest line is computed using a point-to-line distance calculation over all the edges of $`B`$. The distances are then averaged for all the points. This calculation is conducted in two directions to compute $`d_{ER}^E(Y',Y)`$ and $`d_{RE}^E(Y,Y')`$.
#### 2. Edge length scores
More details about the metric will be communicated soon. Consists of a score that quantifies the similarity between the edge length of the estimation and that of the reference. The length of the estimated edges and the reference edges denoted as $`L_{E}`$ and $`L_{R}`$, respectively, are computed by summing over the lengths of the edges of each edge-set.
The edge length score is obtained using a similar strategy to [surface area scores](evaluation.md#L146).
```math
S_l = 1 - | \frac{1 - \alpha (L_{E} / L_{R} + \beta)}{1 + \alpha (L_{E} / L_{R} + \beta)} | .
```
where $`\alpha`$ and $`\beta`$ are two parameters that were set to 2.5 and -0.6, respectively. This score results in a value in [0,1]. Good estimations are expected to have high length scores.
Note: As mentioned in [challenge 3 description](challenge_3.md#challenge-3-recovery-of-feature-edges-in-object-scans), the ground-truth sharp edges were filtered to match the visble parts of the scans. In order to not penalize submissions that were able to estimate sharp edges in the missing parts of the input scans, the edge recovery scores are computed on the filtered edges and the original ones and the best scores are considered.
### Final score
Consists of a combination of the measures evaluating the shape reconstruction and the edge recovery as explained above.
The shape reconstruction score is computed as follows,
```math
S_s = S_a [\Phi_{k_1}(d_{ER}^{shape}(Y',Y)) h_{ER} + \Phi_{k_2}(d_{RE}^{shape}(Y,Y')) h_{RE} ] , \\
```
where $`\Phi_{k_i}(d) = e^{-k_id^2}`$ maps a distance $`d`$ to a score in [0,1]. The parameters $`k_i`$ are chosen according to some conducted baselines.
The edge recovery score is computed as follows:
```math
S_e = S_l [\Phi_{k_1}(d_{ER}^{E}(E',E)) + \Phi_{k_2}(d_{RE}^{E}(E,E')) ] , \\
```
where $`\Phi_{k_i}(d) = e^{-k_id^2}`$ maps a distance $`d`$ to a score in [0,1]. The parameters $`k_i`$ are chosen according to some conducted baselines.
The final score is finally given by,
```math
S = \frac{1}{2} (S_s + S_l) .
```
## References ## References
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment