Multi-layer prototype learning with Dirichlet mixup for open-set EEG recognition

In real-world scenarios, electroencephalograph (EEG)-based brain–computer interface (BCI) systems do not solely receive control signals from the limited classes on which they were trained. They encounter a myriad of both unknown and novel signals. This underscores the necessity for BCI models to focus on the challenge of open-set recognition (OSR), which requires the simultaneous discernment of known samples and exclusion of unknown ones. To address this issue, we introduce a framework for multi-layer prototype learning with Dirichlet mixup (MPL-DM). The MPL-DM framework integrates the concept of prototype learning. By learning prototypes across its multi-layer architecture, the framework achieves a layer-wise ensemble effect within a single network, enhancing the ability to estimate uncertainties. Furthermore, Dirichlet mixup augmentation is incorporated to enrich the training process with synthetic open-set samples, thereby emulating novel or unknown inputs. Rigorous evaluations demonstrated that this approach outperforms conventional methods in handling the intricacies of open-set EEG recognition tasks.