Improving the classification of motor imagery by combining EEG and MEG signals in BCI

WeChat search 《 Brain computer leader 》, Follow article publishing in real time ！

Share an article 2019 Published in INTERNATIONAL JOURNAL OF NEURAL SYSTEMS（ Computer science region 1 District TOP） Of Integrating EEG and MEG Signals to Improve Motor Imagery Classification in Brain-Computer Interface

​

Abstract ： We used a fusion method , This method combines the EEG recorded at the same time （EEG） Magnetoencephalography （MEG） The function of the signal , To improve the brain computer interface based on motor imagination （BCI） The classification performance of . We apply our approach to a set of 15 Healthy subjects , Discovery and α and β Compared with the standard single mode method in the spectral band , The classification performance is significantly improved . in summary , Our findings demonstrate that the multimodal approach is considered to improve noninvasive BCI Complementary tools for .

1.  Preface

Brain-machine interface (BCI) Use the subjects' ability to regulate their brain activity through conscious mental effort , Such as sports imagination (Motor Imagery,MI).BCIs More and more used for control and communication , And the treatment of nervous system diseases . Despite their social and clinical implications , Many engineering challenges remain , From the optimization of control features to the identification of the best psychological strategies , To encode the user's intentions . Besides ,15% To 30% Of users are subjected to a process called “ Brain computer interface blindness ” The phenomenon of , This phenomenon includes not being able to properly control the BCI even after several training sessions .

Brain computer interface (BCI) pays special attention to MI Brain computer interface , Because there are inherent difficulties in producing distinguishable patterns of brain activity . These challenges have seriously affected the MI The availability of BCI , To encourage people to understand MI A deeper understanding of the relevant mechanisms , It also urges people to study new features , To improve the brain computer interface performance of healthy subjects and patients . In the latter case , Hybrid and multimodal methods have been shown to improve overall performance by adding different types of biological signals and neuroimaging data, respectively , Such as near infrared spectroscopy (NIRS) And functional magnetic resonance imaging (fMRI).

ad locum , Let's consider magnetoencephalography (MEG), It has complementary information on source depth and conductivity sensitivity , But there are also radial / Tangential dipole detection . Although previous studies have demonstrated that brain magnetic activity is based on BCI And the feasibility of neural feedback , But with EEG (EEG) The potential benefits of signal binding have not been fully explored .

in fact , Development of portable magnetic intensity sensor based on optical pump magnetometer , This integration can have a real impact . To address this knowledge gap , We considered a group of healthy subjects based on MI High density EEG and MEG signals were recorded simultaneously in the BCI task . then , We propose a matching score fusion method to test the performance of BCI MI Improvement of classification ability .

2.  Materials and methods

2.1 Simultaneous recording E/MEG

15 Healthy subjects ( Age is 28.13 ±4.10 year ,7 Famous woman ), They are all right-handed . No one has a medical or psychological disorder . according to 《 Declaration of Helsinki 》, After explaining the research , Obtain the written informed consent of the subject , And approved by the ethics committee of Paris . All participants received financial compensation at the end of their participation . Use Elekta Neuromag TRIUXR machine ( Include 204 A plane gradiometer and 102 A magnetometer ) and 74 individual EEG The channel system simultaneously recorded MEG and EEG data . The location of EEG electrodes on the scalp follows 10-10 Montage standard .

EEG signal refers to mastoid signal , The grounding electrode is located in the left scapula , The impedance is kept at 20 k following . On average, , The preparation time of the subjects is 1.5 Hours ( That is, explain the scheme 、 Place electrodes 、 Record the EEG sensor position and check the impedance ).M/EEG The data is recorded in the magnetic shielding room , Sampling frequency is 1 kHz, The bandwidth is 0.01-300 Hz. The subjects sat in front of the screen , A distance of 90 centimeter . In order to keep the position of both hands stable , Subjects placed their arms on comfortable supports , Palm up . At the same time, the EMG signals of the subjects' left and right arms were recorded . Bioengineers examined EMG activity with the naked eye , To ensure that the subjects did not move their forearms during recording . We go through Fieldtrip The buffer transmits EEG signals to BCI2000 toolbox46 Conduct brain computer interface conversation .

2.2 BCI agreement

We use one-dimensional 、 Having two goals 、 Right aligned block task , In this task , The subject had to perform a sustained right hand grab to hit the upward target , At the same time, keep still to hit the downward target . Each test has an upward and downward goal , Include the gray vertical bar graph displayed on the right side of the screen , Uniform random distribution in the test .

The experiment is divided into two stages ：

(i) Training : The training phase consists of five consecutive runs , No feedback . In a given test , The first second corresponds to the stimulus interval (ISI), That is, a black screen is presented to the subjects . The goal is in the following 5 second ( from 1 Seconds to 6 second ) Appears and continues on the screen . In the meantime , The subjects must perform the indicated mental tasks .

(ii) test : The test phase consists of six runs and a visual feedback . In a given experiment , The first second corresponds to ISI, The target is presented in the same way as the training phase in the next five seconds . In the last three seconds ( from 3 Seconds to 6 second ), Subjects received a visual feedback to control a cursor ( Here is a ball ) The object of composition , The cursor starts at the middle left of the screen , Move to the right part of the screen at a fixed speed . This gives a fixed communication rate per minute 20 An order . Only the vertical position is controlled by the subject's brain activity , Every time 28 Millisecond update once . The purpose is to hit the target with the ball according to the psychological task of the guidance , That is, the movement imagination of the upward target ; Take a rest towards your goal .

2.3. Signal processing and feature extraction

We considered the activities of EEG and MEG , The latter is controlled by a magnetometer (MAG) And gradiometer (GRAD) Signal composition , Given their physical properties , Can be handled separately .

First , utilize MaxFilter (Elekta Neuromag) Spatial separation of time domain signals for magnetoencephalogram activities (tSSS), Remove environmental noise . All signals are sampled downstream to 250hz, And it is divided into... Corresponding to the target period 5 Second period . To simulate the online scene , There is no artifact removal method . The expert bioengineer visually inspected the traces recorded , To ensure that there are no major artifacts ( Such as MEG jumping , EEG bounce ) There is . After verification , We have preserved all available ages . We calculated for each sensor MI and rest Period power spectrum , The power spectrum is 4 Hz To 40 Hz Between , frequency bin A resolution of 1 Hz.

So , We use a long ellipsoid sequence based on discretization (Slepian sequences) Multi taper frequency conversion , adopt Fieldtrip The tool box is considered a taper . use ±0.5 hz Multi cone spectrum smoothing . At this stage , Each epoch uses a characteristic matrix Mi To represent ,Mi Include the power spectrum value of each pair of sensors and frequency library , The dimensions are 74*36、102*36 and 204*36, about i=EEG、MAG and GRAD. In the training phase , We use a semi-automatic program from the matrix Mi Extract the most relevant features from . First , We focus on the sensor of the moving area on the opposite side of the motion ( See figure in the appendix A.1). such ,EEG、MAG and GRAD The size of the characteristic matrix is 8*36、11*36 and 22*36. secondly , For each selected sensor and frequency library , We are MI A nonparametric cluster based permutation is made between the power spectrum value and the rest time t test . So , We set a statistical threshold ,p < 0.05, Error detection rate correction for multiple comparisons ,500 Permutation .

chart a . 1   Preselected EEG and MEG sensors ( Left sports area ).

Finally, in the standard frequency band b =θ(4-7 Hz),α(8-13 Hz),β(14-29 Hz),γ(30-40 Hz) Take it out internally Nf The most distinguishing feature . This enables us to identify , For each mode i And frequency band b, The best to use during the testing phase ( sensor , Frequency library ) To calculate the characteristics m. therefore , The final feature vector for classification is ：

among Nf = 1,…10. Based on the actual number of features we use in recording sessions (4 To 6 Between ) choice 10 The maximum limit of , Conform to similar rules based on MI Of BCI and EEG Montage related guidelines .

2.4.  classification 、 Fusion and performance evaluation

We are right. Nf Each value of is classified separately . Because the number of features is relatively small , We are based on linear discriminant analysis (LDA) Five fold cross validation is used in the classification of .LDAs It is especially suitable for two types of MI Of BCIs. To integrate information from different modes , We adopt Bayesian fusion method based on weighted average method . Mix with BCI The system is similar to , We have linear combinatorial posterior probabilities pi, pi By each mode i The classification of , With parameters λi weighting ：

In this way , Assign a higher weight to the schema that best classifies the data ( See the picture 1).

chart 1   A classifier fusion method for a given frequency library . Variable pi and λi Represent a posteriori probability and modal respectively i Relevant weight parameters

To evaluate the performance of the classifier , We measured the receiver working characteristics of the calculated values of false positive rate and true positive rate (ROC) curve (AUC) Lower area .AUC The value is usually in 1/2( Opportunity level ) and 1( Perfect classification ) Between . We separate them according to each single pattern (EEG、MAG and GRAD) The results obtained evaluate our fusion method . Besides , We also tested the effect of adding more and more important features .

To statistically compare the results , We will respond accordingly AUC Values are entered into the... Based on nonparametric permutations ANOVA in , The variance consists of two factors : Pattern (EEG, MAG, Gradient and fusion ) And characteristics (Nf = 1,2,…, 10).p < 0.05 The statistical threshold and 5000 Permutations are fixed . Last , We use Tukey Kramer Methods statistical threshold analysis was performed p < 0.05 After the fact analysis . This analysis uses Standards MATLAB and EEGLAB Routines available in the toolbox .

3. result

chart 2 A map showing the total average time frequency associated with the event / Sync (ERD/S) Calculated MI Test phase

, Xtarget The time frequency energy target segment of the sensor signal (1 - 6 S) And the base is the corresponding baseline energy (0 - 1 year ). In all forms , We observed significant changes in α(ERD About equal to −100%) and β Frequency band (ERD About equal to −60%). period ERDs Start appearing after the target appears (t = 1 s), And in the feedback phase (t = 3-6 s) Become stronger . Compared with EEG signals ,ERDs It tends to appear in the early stage of MEG signals .

chart 2 Shown .ERD/S Generalized average time-frequency diagram of . The top panel shows the visual stimuli that occur during the target time period . The dotted line marks the beginning and feedback phase of the goal presentation ( see 2.2 section , Testing phase ). The time-frequency decomposition of the signal passes through 4 Hz To 40 Hz Between Morlet Wavelets get , The central frequency is 1 Hz, The time resolution through the brainstorming toolbox is 3 s. Positive ERD/S The value represents the percentage increase ( That is, neural activity is synchronized ), A negative value represents the percentage reduction ( That is, the neural activity is desynchronized ).

chart 3 It explains the candidate features selected for each mode by semi-automatic program in the training phase . The features obtained from MEG signals tend to focus more on space ( Around the main movement area of the hand ) And frequency ( Mainly in the α Band ). This discovery is related to β belt ERDs Consistent with the lower facts ( See the picture 2).

chart 3   Select spatial and frequency distribution features to classify in each mode . On the left side , Color identification band of nodes ( Blue stands for α Frequency band , Red representative β Frequency band ). The size of the circle is proportional to the number of subjects , The subject takes the specific sensor as the best feature . On the right side , The histogram describes in detail the occurrence of the most frequently selected sensor in each frequency library .

surface 1 Overview of cross modal personal performance . Bold indicates the best result obtained in a certain subject AUCs.

chart 4 stay 15 Of the subjects , For different modes ( Electroencephalogram ,MAG, Gradient and fusion ) And the characteristics of different quantities Nf stay α Band AUC Distribution , The white circle represents the relevant median .

Fusion improves classification performance

In all frequency bands , Modal type significantly affects AUC value ( variance analysis ,p < 10^-3), The number of features has no significant effect (p > 0.05). Obtained by fusion method AUC The value is significantly higher than any other mode (Tukey Kramer after ,p < 0.016), except θ- Band and γ Band , We did not observe any significant improvement in EEG . stay α Band achieves the highest classification performance ( chart 4), Regarding this , We also report a significant interaction between modes and the number of features ( variance analysis ,p = 0.0069). In this case , After fusion AUC The value is significantly higher than EEG、MAG and GRAD Obtained separately AUC value (Tukey Kramer posthoc, p = 4.3* 10^-9, 3.9* 10^-7 and 0.012).

To evaluate the classification performance of each topic , We consider the optimal characteristic number of each mode Nf And with the highest AUC Relevant optimal frequency band . Results show , stay 13 Of the subjects , Compared with the single mode , Fusion for better performance ,AUC The value is 0.55 to 0.85, The relative increment is 1.3% to 50.9%( Average 12.8±6%). Only 3 The fusion effect of subjects was similar ( See table 1).

More specifically , If we compare the fusion effect with the effect of the best single mode , The average improvement range is 4±3%. It is worth noting that , When we compare EEG fusion , The value is 15±17%, EEG fusion is the most commonly used mode in brain computer interface experiments . Interestingly , As with parameters λi The relevant weights are shown in , The contribution of different models to fusion performance varies greatly among subjects ( See the picture 5).

chart 5   The contribution of different models to individual performance . The pie chart shows the results obtained by the fusion method for each mode λi value ( Expressed as a percentage ).

4. result

Improving performance is still one of the most challenging problems in non-invasive BCI systems . High classification performance will allow BCI Effective control and feedback on the topic , This is critical to establishing optimal user machine interaction .BCI Performance depends on several human and technical factors , Including subjects' ability to produce distinguishable brain features ,3 And the robustness of signal processing and classification algorithms . So , We recorded the simultaneous EEG and MEG signals of a group of healthy subjects who performed the brain computer interface task based on motion imaging . Both EEG and MEG show high temporal resolution , And the changes related to sensory movement are well known in the literature , Because of their utility in standard BCI applications .

It is worth noting that ,EEG and MEG Signals are closely related , But they have no effect on radial currents 、 Tangential current 、 The sensitivity of extracellular current and intracellular current is still different . Our fusion method can take advantage of these complementarities to better identify ERD Mechanism , To control BCI. It turns out that , stay α- and β- Band ( In a more limited way ) Got the best AUCs, This is related to sports imagination α- and β- The vibration correlation of the wave band is consistent .

The fusion method shows that , Combine the most important features of each model , On most topics , Reduce subject mental state misclassification ( See table 1). By optimizing the selection of features in each individual , Compared with the average classification, we get 12.8% Separate EEG , Journal and graduate classification ( surface 1), This is to indirectly reduce non-invasive BCIs The phenomenon of illiteracy provides a feasible choice .

In this study , We also explored the characteristics of other frequency bands , Such as γ Band (30-40 Hz). However , And α- and β- Band comparison , The results are marginally improved . Although from intracranial recordings or local field potentials γ- Activity is usually associated with exercise / The activation of sensory function , but γ- The lack of band results can be partly explained by low signal-to-noise ratio and volume conduction effect , This usually affects scalp EEG and MEG Activities . The core of our method is to automatically weight the contribution of each pattern , In an effort to optimize performance .

This is an important aspect , Because the ability to identify features may suddenly change , It depends on many factors , Such as impedance fluctuation or artifact ( For example, isolated EEG electrodes pop up or MEG jumps ) The existence of . under these circumstances , Our fusion method will take into account this transient fluctuation , By using a lower weight in the classification λi To suppress the affected modes . Slower changes may be associated with individual precise control BCI The ability to increase . under these circumstances , Our method will gradually benefit the pattern of spatiotemporal characteristics , Better capture these neural plasticity phenomena .

Interestingly , We noticed a high inter-subject Variability holds that weights ( chart 5), This may be related to the ability of each form to detect potential ERD Different properties of , Further analysis , Maybe in the source space ,84 There is a need to elucidate the neurophysiological changes associated with this aspect of recognition . Although the average AUC Relatively low value , But we noticed that , They vary greatly from individual to individual ( surface 1), And they are close to the values usually obtained in similar experimental settings . Besides , It is worth mentioning that , The subjects were bci Childish , And no preprocessing is applied , The purpose is to simulate real life scenes . therefore , Although a proper preprocessing is likely to improve accuracy in each single mode , Our goal is to assess improvements in the worst conditions .

Final , our 15 Of the subjects 13 People show improvement after classifier fusion . in summary , These results demonstrate the simultaneous use of E/MEG Potential advantages of signal to improve brain computer interface performance . By using a fairly simple classifier (LDA), We can include fewer specific features involving motor related neural mechanisms in the classification , Such as α and β Band ERD. More complex methods using the entire feature space , Such as support vector machines85 And Riemannian geometry ,86 And alternative fusion strategies , If increase 、 Voting or stacking strategy ,55 Still in source space , Improve the spatial resolution and identify the time-frequency characteristics of classified accounts , It can be further evaluated in practical application , Use their power .

Last , The important thing is to pay attention , We tested our fusion method offline by analyzing previously recorded data . To evaluate the feasibility of online applications , We estimate that in 500 ms Of epoch in , When Nf = 5 when , Calculating characteristics 、 The time required for classification and determination of fusion parameters is about 20 ms. This value is actually compatible with the current online settings , Use a similar time window , Every time 28 Update feedback every millisecond .

5. Conclusion

Our results show that , Integrating the information of EEG and MEG synchronization signals can improve the performance of BCI .E/MEG Multimodal BCI It may become an effective method to improve the reliability of brain computer interaction , But much of the progress will depend on MEG Miniaturization of scanner , At present, a magnetic shielding room is required (MSR) And sensors cooled by a cryogenic system . Recent efforts have proposed miniaturized and non cryogenic magnon sensors , And avoid using MSRs, This is expected to improve the portability of magnons and promote multimodality BCI To provide practical solutions for the development of .