Algorithm Challenge
What is it
The Algorithm Challenge is a competition to advance methods that reconstruct motor unit spike trains from high-density surface EMG (HDsEMG). It consists of two independent tasks and two sequential phases.
To participate, register on Codabench, then navigate to the Algorithm challenge page!
Tasks
Competitors may participate in one or both tasks:
- Isometric task. Well-studied, stationary conditions for which multiple neural source separation methods currently exist.
- Dynamic task. Less studied, non-stationary conditions in which algorithmic performance remains limited, motivating the development of novel approaches and interactive exchange of ideas.
Each task yields an independent leaderboard.
Phases
Familiarization Phase
Running in parallel with the Data Challenge, algorithm developers are provided with test data for the Isometric Task and the Dynamic Task to build, test, and optimize their motor unit identification algorithms.
- Participants can compete in both tasks or in only one task
- Competitors upload their predictions (motor unit spike trains) through the algorithm submission portal
- A preliminary leaderboard is provided as feedback
- Results from baseline algorithms (coming soon) are provided for reference
- Participation in the Familiarization phase is encouraged by the organizers
Showdown Phase
The main algorithm competition conducted using a community-based collection of experimental and simulated HDsEMG datasets for the Isometric Task and the Dynamic Task.
- Submission of predicted motor unit spike trains for each recording
- To be eligible for awards, competitors need to share their code openly (e.g., using GitHub); submissions based on proprietary code are not eligible for awards
- The final leaderboards for the Isometric and Dynamic tasks will be based solely on performance during the Showdown Phase and will be publicly released
See the Submission and Registration page for details on how to submit predictions.
Leaderboard scoring
The data collection for the Isometric Task (Familiarization collection and Showdown collection) is a mixture of simulated HDsEMG recordings with univocal ground-truth labels and experimental data with expert-curated — yet incomplete — labels.
The data collection for the Dynamic Task (Familiarization collection and Showdown collection) is a mixture of simulated HDsEMG recordings with univocal ground-truth labels and perturbed isometric data with expert-curated — yet incomplete — labels.
For the scoring, we perform the following steps:
- (1) Remove duplicate spike trains: A duplicate motor unit is detected if two spike trains share at minimum 30 percent of common spikes. In the case of duplicates, we will keep the first unit and reject all other units.
- (2) Match predicted and ground truth spike trains: Compute for all pairs of predicted and ground truth spike trains the F1-score and match labels by solving a linear sum assignment problem. Matches require a minimum F1-score of 30 percent.
- (3) Label-based score: The label-based score of each recording is the weighted sum of all accepted units. For all units with an F1 score above 50 percent, the weight is the F1 score. Otherwise, the weights are zero.
For all recordings with incomplete labels, we additionally perform the following steps for obtaining model-based confidence scores for unmatched sources:
- (4) Supervised Fitting of a CBSS Model: We use a convolutive blind source separation model (extension factor: 12) and learn the unmixing weights given the predicted spike labels. As there are multiple delayed copies of the same spike train, we will consider all delays within a window of plus/minus 0.01 seconds and use the unmixing weights that yield the highest silhouette-like score. Note: For perturbed isometric data in the dynamic task, we make use of the unperturbed isometric data.
- (5) Additional metrics: To further evaluate the plausibility of the predicted sources, we calculate the pulse-to-noise ratio and the coefficient of dispersion (median absolute deviation divided by the median) of the interspike intervals.
- (6) Bad source rejection: All sources with fewer than 25 spikes or a silhouette-like score below 0.85 are rejected.
- (7) Map quality metrics to confidence scores: To map the quality metrics (silhouette score, pulse-to-noise ratio, and the coefficient of dispersion of the interspike intervals) to a normalized confidence score between zero and one, we use a data-driven mapping given in the following table.
- (8) Model-based score: To get the model-based score, we compute for each predicted motor unit the mean value of the three metric-based confidence scores and sum these values.
For perturbed isometric data with incomplete labels in the dynamic task, we perform one additional step to detect missing spikes due to insufficient tracking of the underlying non-stationarity:
- (9) Weighting: We compute for each unmatched source a pseudo-Recall between the predicted spikes and the model-based spikes obtained from the fitted CBSS model. This pseudo-Recall score is used as a multiplicative weighting factor in the model-based score (per unit).
The recording score is the label-based score plus (if the labels are incomplete) the model-based score. Finally, the submission score is the average of the recording scores for the respective task (missing prediction files yield a recording score of zero).
The scoring system implementation is publicly available on GitHub.
| Confidence score | Silhouette score | Pulse-to-Noise ratio | CoD-ISI |
|---|---|---|---|
| 1.0 | 0.95 ≤ SIL | 37.1 ≤ PNR | 0.05 ≤ CoD-ISI < 0.197 |
| 0.75 | 0.925 ≤ SIL < 0.95 | 33.9 ≤ PNR 37.1 | 0.197 ≤ CoD-ISI < 0.372 |
| 0.5 | 0.85 ≤ SIL < 0.925 | 28.0 ≤ PNR < 33.9 | 0.372 ≤ CoD-ISI < 0.73 |
| 0.0 | SIL < 0.85 | PNR < 28.0 | CoD-ISI < 0.05 or CoD-ISI > 0.73 |
Showdown phase
During the showdown phase, we will use the scoring implementation described and linked above. At the end of the showdown phase, we may modify the confidence score assignment to unmatched sources (see table above) by using the empirical distribution of SIL, PNR, and CoD-ISI scores from accepted sources across all submissions, as well as update the label set of the experimental data by manually curating unmatched units identified by the participants.
Awards
All teams receive recognition on a permanent leaderboard (per task). The top 5 teams per task (Isometric and Dynamic) will be invited to share their solutions in a special issue of the Journal of Electromyography and Kinesiology. All winning teams will also be invited to share their work in a collaborative benchmarking paper in the same special issue!