wiki.neuro.tu-berlin.de

Background

Context and previous studies

Integrating user-state biofeedback is vital for providing more immersive, adaptive and functional VR experiences[1][2][3]. VR headsets would be an ideal use-case, especially when paired with recent advances in dry and wireless EEG elecrodes[4][5][6][7][8], as well as in motion artifact suppression[9][10][11][12][13].

Prior Studies

Active or Reactive vs Passive Modulation

On one hand, there already have been several studies, who focused on modulation. The majority chose to focus on active or reactive modulation[14], where the participant tries to directly control or influence the environment either intentionally or reactively and usually this has been carried out in a setting, where the participant remained stationary in physical space[15][16][17][18][19]. However, this kind of setup requires significant user training to perform at acceptable levels and rely on unnatural or obtrusive sensory stimuli. Furthermore, these designs exhibit performance issues that limit practical, long-term use[20][21][22].

On the other hand, implicit or passive BCI control, i.e. the user's cognitive or affective state is passively monitored and used to affect some auxiliary aspect of interaction[14:1][23][24][25][26]. Moreover, they can also be designed to be less sensitive to BCI decoding errors and could potentially also be less noticeable and distracting, since the user does not have to directly control, think or act. This basically amounts to trading off direct and potentially intentional control for indirect complementary and restricted control, possibly resulting in better immersion.

Tasks, Difficulty and Cognitive Workload

Prior studies atteempted to classify different cognitive tasks using brain activity, e.g. rest vs. mental imagery(mental math or object rotation). The number of tasks and difficulty can be altered to produce detactable changes in cognitive state, as demonstrated in both EEG[27][28][29] and fNIRS[30][31][32]. Other studies have further investigated changees in brain activity with respect to changes in cognitive workload during task performance using EEG[33][34][35][36][37][38][39][40][41][42]. It has also been shown that cognitive workload models trained on one task could be transferred to others[43]

Applications of Passive modulation

There exists applications in studies that used passive neurofeedback of EEG or fNIRS. Here is a list of some studies of such applications for the following scenarios:

Furthermore, reviews on applications of BCI for VR and video games has been done by [48][49][50].

Detecting Cognitive Workload

EEG Lobes
Figure 1 - EEG Channels and Lobes(Frontal Lobe, Parietal Lobe, Temporal Lobe, Occipital Lobe)

The aforementioned EEG-based studies largely utilize various combinations of the traditional power spectral bands over frontal, central, and parietal locations. Numerous studies indicated that the fronto-parietal network shows signs of a decrease in α power with increasing task demands, while θ power is positively correlated with increasing task demands[51][52][34:1]. Other studies suggest β behaves like α but that may be due to motor activity required by the tasks[53][41:1]. Due to the limitation of scalp EEG, γ activity has been less frequently reported in relation to cognitive workload. Aside from that, there have been a further discoveries, such as:

 

  1. Bisson, E., Contant, B., Sveistrup, H., and Lajoie, Y. (2007). Functional balance and dual-task reaction times in older adults are improved by virtual reality and biofeedback training. Cyberpsychol. Behav. 10, 16–23. doi: 10.1089/cpb.2006.9997 ↩︎

  2. Lobel, A., Gotsis, M., Reynolds, E., Annetta, M., Engels, R., and Granic, I. (2016). “Designing and utilizing biofeedback games for emotion regulation: the case of nevermind,” in CHI EA '16 Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, CA), 1945–1951. doi: 10.1145/2851581.2892521 ↩︎

  3. Cipresso, P., Giglioli, I. A. C., Raya, M. A., and Riva, G. (2018). The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature. Front. Psychol. 9:2086. doi: 10.3389/fpsyg.2018.02086 ↩︎

  4. Wang, S., Gwizdka, J., and Chaovalitwongse, W. A. (2016). Using wireless EEG signals to assess memory workload in then-back task. IEEE Trans. Hum. Mach. Syst. 46, 424–435. doi: 10.1109/THMS.2015.2476818 ↩︎

  5. Zander, T. O., Andreessen, L. M., Berg, A., Bleuel, M., Pawlitzki, J., Zawallich, L., et al. (2017). Evaluation of a dry EEG system for application of passive brain-computer interfaces in autonomous driving. Front. Hum. Neurosci. 11:78. doi: 10.3389/fnhum.2017.00078 ↩︎

  6. de Camp, N. V., Kalinka, G., and Bergeler, J. (2018). Light-cured polymer electrodes for non-invasive EEG recordings. Sci. Rep. 8:14041. doi: 10.1038/s41598-018-32304-6 ↩︎

  7. Lee, S., Shin, Y., Kumar, A., Kim, M., and Lee, H.-N. (2018). Dry electrode-based fully isolated EEG/fNIRS hybrid brain-monitoring system. IEEE Trans. Bio-med. Eng. 66, 1055–1068. doi: 10.1109/TBME.2018.2866550 ↩︎

  8. Kam, J. W. Y., Griffin, S., Shen, A., Patel, S., Hinrichs, H., Heinze, H.-J., et al. (2019). Systematic comparison between a wireless EEG system with dry electrodes and a wired EEG system with wet electrodes. NeuroImage 184, 119–129. doi: 10.1016/j.neuroimage.2018.09.012 ↩︎

  9. Gwin, J. T., Gramann, K., Makeig, S., and Ferris, D. P. (2010). Removal of movement artifact from high-density EEG recorded during walking and running. J. Neurophysiol. 103, 3526–3534. doi: 10.1152/jn.00105.2010 ↩︎

  10. Daly, I., Scherer, R., Billinger, M., and Müller-Putz, G. (2015). Force: fully online and automated artifact removal for brain-computer interfacing. IEEE Trans. Neural Syst. Rehabil. Eng. 23, 725–736. doi: 10.1109/TNSRE.2014.2346621 ↩︎

  11. Kline, J. E., Huang, H. J., Snyder, K. L., and Ferris, D. P. (2015). Isolating gait-related movement artifacts in electroencephalography during human walking. J. Neural Eng. 12:046022. doi: 10.1088/1741-2560/12/4/046022 ↩︎

  12. Snyder, K. L., Kline, J. E., Huang, H. J., and Ferris, D. P. (2015). Independent component analysis of gait-related movement artifact recorded using EEG electrodes during treadmill walking. Front. Hum. Neurosci. 9:639. doi: 10.3389/fnhum.2015.00639 ↩︎

  13. Arad, E., Bartsch, R. P., Kantelhardt, J. W., and Plotnik, M. (2018). Performance-based approach for movement artifact removal from electroencephalographic data recorded during locomotion. PLoS ONE 13:e0197153. doi: 10.1371/journal.pone.0197153 ↩︎

  14. Zander, T. O., Gaertner, M., Kothe, C., and Vilimek, R. (2010). Combining eye gaze input with a brain–computer interface for touchless human–computer interaction. Int. J. Hum. Comput. Interact. 27, 38–51. doi: 10.1080/10447318.2011.535752 ↩︎ ↩︎

  15. Leeb, R., Friedman, D., Muller-Putz, G.-R., Scherer, R., Slater, M., and Pfurtscheller, G. (2007). Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegic. Comput. Intell. Neurosci. 5, 117–128. doi: 10.1155/2007/79642 ↩︎

  16. Scherer, R., Lee, F., Schlogl, A., Leeb, R., Bischof, H., and Pfurtscheller, G. (2008). Toward self-paced brain? computer communication: navigation through virtual worlds. IEEE Trans. Biomed. Eng. 55:2. doi: 10.1109/TBME.2007.903709 ↩︎

  17. Royer, A. S., Doud, A. J., Rose, M. L., and He, B. (2010). EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies. IEEE Trans. Neural Syst. Rehabil. Eng. 18, 581–589. doi: 10.1109/TNSRE.2010.2077654 ↩︎

  18. Velasco-Alvarez, F., Ron-Angevin, R., and Blanca-Mena, M. J. (2010). Free virtual navigation using motor imagery through an asynchronous brain-computer interface. Presence 19, 71–81. doi: 10.1162/pres.19.1.71 ↩︎

  19. Doud, A. J., Lucas, J. P., Pisansky, M. T., and He, B. (2011). Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface. PLoS ONE 6:e26322. doi: 10.1371/journal.pone.0026322 ↩︎

  20. Lotte, F., Renard, Y., and Lecuyer, A. (2008). “Self-paced brain-computer interaction with virtual worlds: a quantitative and qualitative study out of the lab,” in 4th International Brain Computer Interface Workshop and Training Course 2008 (Graz). ↩︎

  21. Lotte, F., Van Langhenhove, A., Lamarche, F., Ernest, T., Renard, Y., Arnaldi, B., et al. (2010). Exploring large virtual environments by thoughts using a brain-computer interface based on motor imagery and high-level commands. Presence 19, 54–70. doi: 10.1162/pres.19.1.54 ↩︎

  22. Ron-Angevin, R., and Díaz-Estrella, A. (2009). Brain-computer interface: changes in performance using virtual reality techniques. Neurosci. Lett. 449, 123–127. doi: 10.1016/j.neulet.2008.10.099 ↩︎

  23. Brouwer, A.-M., Zander, T. O., van Erp, J. B. F., Korteling, J. E., and Bronkhorst, A. W. (2015). Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls. Front. Neurosci. 9:136. doi: 10.3389/978-2-88919-613-5 ↩︎

  24. Unni, A., Ihme, K., Jipp, M., and Rieger, J. W. (2017). Assessing the driver's current level of working memory load with high density functional near-infrared spectroscopy: a realistic driving simulator study. Front. Hum. Neurosci. 11:167. doi: 10.3389/fnhum.2017.00167 ↩︎

  25. Horvat, M., Dobrinić, M., Novosel, M., and Jerčić, P. (2018). “Assessing emotional responses induced in virtual reality using a consumer EEG headset: a preliminary report,” in 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) (Opatija: IEEE). doi: 10.23919/mipro.2018.8400184 ↩︎

  26. Ihme, K., Unni, A., Zhang, M., Rieger, J. W., and Jipp, M. (2018). Recognizing frustration of drivers from face video recordings and brain activation measurements with functional near-infrared spectroscopy. Front. Hum. Neurosci. 12:327. doi: 10.3389/fnhum.2018.00327 ↩︎

  27. Ruchkin, D. S., Johnson, R., Canoune, H., and Ritter, W. (1991). Event-related potentials during arithmetic and mental rotation. Electroencephalogr. Clin. Neurophysiol. 79, 473–487. doi: 10.1016/0013-4694(91)90167-3 ↩︎

  28. Ryu, K., and Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int. J. Indus. Ergon. 35, 991–1009. doi: 10.1016/j.ergon.2005.04.005 ↩︎

  29. So, W. K. Y., Wong, S. W. H., Mak, J. N., and Chan, R. H. M. (2017). An evaluation of mental workload with frontal EEG. PLoS ONE 12:e0174949. doi: 10.1371/journal.pone.0174949 ↩︎

  30. Power, S. D., Falk, T. H., and Chau, T. (2010). Classification of prefrontal activity due to mental arithmetic and music imagery using hidden markov models and frequency domain near-infrared spectroscopy. J. Neural Eng. 7:26002. doi: 10.1088/1741-2560/7/2/026002 ↩︎

  31. Power, S. D., Kushki, A., and Chau, T. (2012). Intersession consistency of single-trial classification of the prefrontal response to mental arithmetic and the no-control state by nirs. PLoS ONE 7:e37791. doi: 10.1371/journal.pone.0037791 ↩︎

  32. Herff, C., Heger, D., Putze, F., Hennrich, J., Fortmann, O., and Schultz, T. (2013b). Classification of mental tasks in the prefrontal cortex using fNIRS. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2013, 2160–2163. doi: 10.1109/EMBC.2013.6609962 ↩︎

  33. Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., et al. (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat. Space Environ. Med. 78, B231–B244 ↩︎

  34. Brouwer, A.-M., Hogervorst, M. A., van Erp, J. B. F., Heffelaar, T., Zimmerman, P. H., and Oostenveld, R. (2012). Estimating workload using EEG spectral power and ERPs in the n-back task. J. Neural Eng. 9:045008. doi: 10.1088/1741-2560/9/4/045008 ↩︎ ↩︎

  35. Gerjets, P., Walter, C., Rosenstiel, W., Bogdan, M., and Zander, T. O. (2014). Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach. Front. Neurosci. 8:385. doi: 10.3389/fnins.2014.00385 ↩︎

  36. Hogervorst, M. A., Brouwer, A.-M., and van Erp, J. B. (2014). Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Front. Neurosci. 8:322. doi: 10.3389/fnins.2014.00322 ↩︎

  37. Mühl, C., Jeunet, C., and Lotte, F. (2014). EEG-based workload estimation across affective contexts. Front. Neurosci. 8:114. doi: 10.3389/fnins.2014.00114 ↩︎

  38. Ewing, K. C., Fairclough, S. H., and Gilleade, K. (2016). Evaluation of an adaptive game that uses EEG measures validated during the design process as inputs to a biocybernetic loop. Front. Hum. Neurosci. 10:223. doi: 10.3389/fnhum.2016.00223 ↩︎

  39. Schultze-Kraft, M., Dähne, S., Gugler, M., Curio, G., and Blankertz, B. (2016). Unsupervised classification of operator workload from brain signals. J. Neural Eng. 13:036008. doi: 10.1088/1741-2560/13/3/036008 ↩︎

  40. Grissmann, S., Faller, J., Scharinger, C., Spüler, M., and Gerjets, P. (2017a). Electroencephalography based analysis of working memory load and affective valence in an n-back task with emotional stimuli. Front. Hum. Neurosci. 11:616. doi: 10.3389/fnhum.2017.00616 ↩︎

  41. Scharinger, C., Soutschek, A., Schubert, T., and Gerjets, P. (2017). Comparison of the working memory load in -back and working memory span tasks by means of EEG frequency band power and p300 amplitude. Front. Hum. Neurosci. 11:6. doi: 10.3389/fnhum.2017.00006 ↩︎ ↩︎

  42. Pergher, V., Wittevrongel, B., Tournoy, J., Schoenmakers, B., and Van Hulle, M. M. (2018). N-back training and transfer effects revealed by behavioral responses and EEG. Brain Behav. 8:e01136. doi: 10.1002/brb3.1136 ↩︎

  43. Baldwin, C. L., and Penaranda, B. N. (2012). Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification. NeuroImage 59, 48–56. doi: 10.1016/j.neuroimage.2011.07.047 ↩︎

  44. Muhl, C., Gurkok, H., Bos, P.-O. D., Thurlings, M. E., Scherffig, L., Duvinage, M., et al. (2010). Bacteria hunt. J. Multimodal Interfaces 4, 11–25. doi: 10.1007/s12193-010-0046-0 ↩︎

  45. Bos, D. P. O., Reuderink, B., van de Laar, B., Gürkök, H., Mühl, C., Poel, M., et al. (2010). “Brain-computer interfacing and games,” in Brain-Computer Interfaces. Human-Computer Interaction Series, eds D. S. Tan and A. Nijholt (London: Springer), 149–178 ↩︎

  46. Girouard, A., Solovey, E. T., and Jacob, R. J. (2013). Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. Int. J. Auton. Adapt. Commun. Syst. 6, 26–44. doi: 10.1504/IJAACS.2013.050689 ↩︎

  47. Zander, T. O., and Kothe, C. (2011). Towards passive brain-computer interfaces: applying brain? computer interface technology to human-machine systems in general. J. Neural Eng. 8:025005. doi: 10.1088/1741-2560/8/2/025005 ↩︎

  48. Lécuyer, A., Lotte, F., Reilly, R. B., Leeb, R., Hirose, M., and Slater, M. (2008). Brain-computer interfaces, virtual reality, and videogames. Computer 41, 66–72. doi: 10.1109/MC.2008.410 ↩︎

  49. Lotte, F., Faller, J., Guger, C., Renard, Y., Pfurtscheller, G., Lécuyer, A., et al. (2013). Combining BCI With Virtual Reality: Towards New Applications and Improved BCI. Berlin; Heidelberg: Springer. ↩︎

  50. Kerous, B., Skola, F., and Liarokapis, F. (2018). EEG-based BCI and video games: a progress report. Virtual Real. 22, 119–135. doi: 10.1007/s10055-017-0328-x ↩︎

  51. Sauseng, P., Klimesch, W., Schabus, M., and Doppelmayr, M. (2005). Fronto-parietal EEG coherence in theta and upper alpha reflect central executive functions of working memory. Int. J. Psychophysiol. 57, 97–103. doi: 10.1016/j.ijpsycho.2005.03.018 ↩︎

  52. Sauseng, P., Griesmayr, B., Freunberger, R., and Klimesch, W. (2010). Control mechanisms in working memory: a possible function of EEG theta oscillations. Neurosci. Biobehav. Rev. 34, 1015–1022. doi: 10.1016/j.neubiorev.2009.12.006 ↩︎

  53. Pesonen, M., Hämäläinen, H., and Krause, C. M. (2007). Brain oscillatory 4-30 Hz responses during a visual n-back memory task with varying memory load. Brain Res. 1138, 171–177. doi: 10.1016/j.brainres.2006.12.076 ↩︎

  54. Fitzgibbon, S. P., Pope, K. J., Mackenzie, L., Clark, C. R., and Willoughby, J. O. (2004). Cognitive tasks augment gamma EEG power. Clin. Neurophysiol. 115, 1802–1809. doi: 10.1016/j.clinph.2004.03.009 ↩︎

  55. Tallon-Baudry, C., Bertrand, O., Peronnet, F., and Pernier, J. (1998). Induced γ-band activity during the delay of a visual short-term memory task in humans. J. Neurosci. 18, 4244–4254. doi: 10.1523/JNEUROSCI.18-11-04244.1998 ↩︎

  56. Roux, F., and Uhlhaas, P. (2014). Working memory and neural oscillations: alpha-gamma versus theta-gamma codes for distinct wm information? Trends Cogn. Sci. 18, 16–25. doi: 10.1016/j.tics.2013.10.010 ↩︎