update evaluation doc

doc/evaluation.md

@@ -19,8 +19,11 @@ Consist of two directed distances:
 
 These distances are inspired from [1] but have been adpated to fit the problem at hand.
 The directed distance $`d_{AB}`$ between meshes $`A`$ and $`B`$ is
-approximated in practice by sampling points on $`A`$ and computing their
-distances to the nearest triangles in mesh $`B`$.
+approximated in practice by sampling points on $`A`$ and computing their distances to the nearest triangles in mesh $`B`$.
+
+
+<img align="right" height="200" src="data/points12-example.png">
+These points are sampled according to the area of the triangles. An example of these points is given in this figure: 
 
 The directed distances $`d_{RE}`$ and $`d_{ER}`$ are given by,
 ```math
@@ -46,7 +49,7 @@ This results in two shape distance values ($`d_{ER}^{shape}`$, $`d_{RE}^{shape}`
 Good estimations are expected to have low shape and texture distance values.
 
 ### 2. Surface hit-rates
-ith the point-to-triangle distance used above.
+
 Consist of two rates that are computed in two directions:
 
 1. $`h_{ER}`$ computed from estimation to reference
@@ -61,7 +64,7 @@ Let us consider:
 - $`H_{AB}`$: number of points of the source mesh $`A`$ that hit the target $`B`$
 - $`M_{AB}`$: number of points of the source mesh $`A`$ that miss the target $`B`$.
 
-The hit-rate from $`A`$ to $`B`ith the point-to-triangle distance used above.$ is then given by,
+The hit-rate from $`A`$ to $`B`$  is then given by,
 ```math
 h_{AB} = \frac{H_{AB}}{H_{AB} + M_{AB}} .
 ```
@@ -82,10 +85,10 @@ These areas are then normalized as follows,
 The area score  $`S_a`$ is then given by,
 
 ```math
-S_a = 1 - | \bar{A_{R}} - \bar{A_{E}} | .
+S_a = 1 - | \bar{A_{R}} - \bar{A_{E}} |   = 1 - |  \frac{1 - A_{E} / A_{R}}{1 + A_{E} / A_{R}}  | .
 ```
 
-This score results in a value in [0,1]. Good estimations are expected to have high area scores.
+This score results in a value in [0,1]. Good estimations are expected to have high area scores. It is important to note that such score is not symmetric: estimations with larger triangle areas than references are less penalized than those with smaller triangle areas than references. 
 
 
 ### Final score
@@ -108,7 +111,7 @@ The final score is finally given by,
 S = \frac{1}{2} S_a(S_s + S_t) .
 ```
 
-Note that challenge 1 is evaluated on specific regions of the body mesh. The head and hands are ignored, rest of the body is considered. In challenge 2, the full mesh is considered for evaluation.  
+Note that challenge 1 is evaluated on specific regions of the body mesh. The head and hands are ignored, rest of the body is considered. In this case, masks are computed on the reference and included in the computation of the scores. In challenge 2, the full mesh is considered for evaluation. 
 
 
 
@@ -125,11 +128,94 @@ Notation:
 
 The quality of the estimations, $`Y'`$ and $`E'`$, are evaluated quantitatively with respect to the ground-truth, $`Y`$ and $`E`$, respectively, using a metric evaluating:
 
-- the shape reconstruction of the corrupted scans `X` with respect to CAD models `Y`.
-- the recovery of the sharp edges with respect to ground-truth edges `E`.
+- Shape reconstruction: the shape reconstruction of the corrupted scans `X` with respect to CAD models `Y`.
+- Edge recovery: the recovery of the sharp edges with respect to ground-truth edges `E`.
+
+### Shape reconstruction 
+
+The shape reconstruction is evaluated on  $`Y'`$ with respect to the corresponding CAD model $`Y`$ using a similar formulation to SHARP [challenges 1 and 2](evaluation.md#quantitative-evaluation-of-challenges-12). 
+
+The first subscore ['surface-to-surface distances'](evaluation.md#1-surface-to-surface-distances) only involves the shape scores of the meshes ( $`Y'`$,  $`Y`$) by computing  ($`d_{ER}^{shape}`$, $`d_{RE}^{shape}`$) with the same manner described [here](evaluation.md#1-surface-to-surface-distances).  <img align="right" height="100" src="data/points3-example.png">
+In order to better reflect the shape reconstrution around the sharp edges on the shape score, additional points are sampled on the triangles nearby the sharp edges. An example of these additional points is shown in this figure:
+
+
+
+The hit-rates [hit-rates](evaluation.md#2-surface-hit-rates) and the [surface area scores](evaluation.md#3-surface-area-scores) are also computed following a similar manner as described for [challenges 1 and 2](evaluation.md#quantitative-evaluation-of-challenges-12). The only difference consists of making the surface area scores more symmetric than in challenges 1 and 2 where the focus was on shape completion. In particular, the shape area score is computed as follows: 
+
+```math
+S_a  = 1 - |  \frac{1 - \alpha (A_{E} / A_{R} + \beta)}{1 + \alpha (A_{E} / A_{R} + \beta)}  | .
+```
+
+where $`\alpha`$ and $`\beta`$ are two parameters allowing the control of the symmetry of the score function. The values of these parameters were set to $`\alpha=2.6`$ and $`\beta = -0.6`$.
+
+Finally, the three subscores are combined as described [here](evaluation.md#final-score) to obtain a final shape reconstuction score $`S_s`$.
+
+ <img align="right" height="100" src="data/innerparts.png">
+Note that the ground-truth CAD models $`Y`$ might contain some inner-parts which are missing in the provided input scans $`X`$. These parts are not visible and cannot be scanned. Consequently, these parts are masked and not included in computing the shape scores. An example of these parts and the masked version of the CAD model is shown in this figure: 
+
+
+### Edge recovery 
+
+In addition to the shape reconstruction, the estimated sharp edges $`E'`$ are evaluated with respect to the ground-truth sharp edges $`E`$. The evaluation of the edges is conducted using two criterions: 
+
+#### 1. Edge-to-edge distances
+
+
+Consist of two directed distances:
+
+1. $`d_{ER}^E`$ is computed from the estimation to the reference
+2. $`d_{RE}^E`$ is computed from the reference to the estimation.
+
+The directed distance $`d_{AB}^E`$ between edges $`A`$ and $`B`$ is
+approximated in practice by sampling points on $`A`$ and computing their distances to the nearest lines in edges $`B`$. 
+<img align="right" height="100" src="data/pointsedges-example.png">
+The points are sampled according to the length of the edges. An example of these points is given in this figure (top: sampled points; bottom: original edge lines): 
+
+For each sampled point in $`A`$, the euclidean distance to the nearest line is computed using a point-to-line distance calculation over all the edges of $`B`$. The distances are then averaged for all the points. This calculation is conducted in two directions to compute $`d_{ER}^E(Y',Y)`$ and $`d_{RE}^E(Y,Y')`$. 
+
+
+#### 2. Edge length scores
 
-More details about the metric will be communicated soon. 
+Consists of a score that quantifies the similarity between the edge length of the estimation and that of the reference. The length of the estimated edges and the reference edges denoted as $`L_{E}`$ and $`L_{R}`$, respectively, are computed by summing over the lengths of the edges of each edge-set.
 
+The edge length score is obtained using a similar strategy to [surface area scores](evaluation.md#L146).
+
+```math
+S_l  = 1 - |  \frac{1 - \alpha (L_{E} / L_{R} + \beta)}{1 + \alpha (L_{E} / L_{R} + \beta)}  | .
+```
+
+where $`\alpha`$ and $`\beta`$ are two parameters that were set to 2.5 and -0.6, respectively. This score results in a value in [0,1]. Good estimations are expected to have high length scores. 
+
+
+Note: As mentioned in [challenge 3 description](challenge_3.md#challenge-3-recovery-of-feature-edges-in-object-scans), the ground-truth sharp edges were filtered to match the visble parts of the scans. In order to not penalize submissions that were able to estimate sharp edges in the missing parts of the input scans, the edge recovery scores are computed on the filtered edges and the original ones and the best scores are considered. 
+
+
+### Final score
+
+
+Consists of a combination of the measures evaluating the shape reconstruction and the edge recovery as explained above. 
+
+The shape reconstruction score is computed as follows,
+
+```math
+S_s = S_a [\Phi_{k_1}(d_{ER}^{shape}(Y',Y)) h_{ER} + \Phi_{k_2}(d_{RE}^{shape}(Y,Y')) h_{RE} ] , \\
+```
+
+where $`\Phi_{k_i}(d) = e^{-k_id^2}`$ maps a distance $`d`$ to a score in [0,1]. The parameters $`k_i`$ are chosen according to some conducted baselines.
+
+The edge recovery score is computed as follows: 
+
+```math
+S_e = S_l [\Phi_{k_1}(d_{ER}^{E}(E',E)) + \Phi_{k_2}(d_{RE}^{E}(E,E')) ] , \\
+```
+
+where $`\Phi_{k_i}(d) = e^{-k_id^2}`$ maps a distance $`d`$ to a score in [0,1]. The parameters $`k_i`$ are chosen according to some conducted baselines.
+
+The final score is finally given by,
+
+```math
+S = \frac{1}{2} (S_s + S_l) .
+```
 
 ## References