covariance map

The true potential of noninvasive neuroengineering is still waiting to be unlocked. Unfortunately, no existing implementation of a noninvasive neurotechnology - one that doesn’t break the skin - yet provides the value required of a consumer facing product. This shouldn’t stop you from pushing forward, but in the following methods, let’s examine the common approaches, and where boundaries and limitations lie.

Table of Contents

  1. Common neural signal features
  2. Signal recording
  3. Signal processing overview

I. Common neural signal features

Not designed for interfacing

The brain wasn’t designed for interfacing. Neurons, having only the evolutionary pressure to communicate with each other electrochemically, have no particular reason to interact with our electronics. Fundamentally, neurons don’t leak a lot of information to the exterior extracellular matrix, and any particular leakage is subject to the inverse square rule (or worse) like most physical processes.

This means that most non-invasively detectible effects occur over very large, aggregate spatial and temporal regions. You are always trading off reliability of detection with resolution and transfer bitrate.

Signal features comparison

Signal featureCommon useLimitations
P300 familiarity signalfamiliarity of imagesneeds repeated exposure, false positives
typing on a keyboardaccomplished by flashing the letters; slow, few letters per minute max achieved
δ-band activity/SCPcursor movementsvery slow cursor modulation/bitrate (tens of seconds)
β-band activity; α/β ratioalertness, awakeness, focuspredominantly modulated by eyes-openess; alertness highly variable between individuals
eyes open or closedlag of seconds to transition
focus levelmuch less accurate than pupil size visualized through a webcam, highly variable between individuals
sensorimotor: μ-band (β-band subset)arm/hand/finger movementcan detect movement onset; high-variance trial-to-trial, slow (seconds)
speechcan detect speech onset; distinguishing speech content extremely difficult
sensorimotor: local motor potentialhand/finger positionhigh-variance trial-to-trial, person-to-person, not well replicated
VEPstyping on keyboardPerson-to-person variability. In persons working, highest transfer bitrate. Requires pre-arranged and accurately-timed stimuli, usually flashing lights. Can be epilepsy trigger.
multiple-choice
Error potentialsmismovement, misspeech, typing correctionPerson-to-person variability. Usually in the μ and β-bands. Aggregate effect, noisy/error-prone to use on individual trials.
EMG activationcursor movementhigh SNR, easy to detect, easy to use; but not particularly impressive, less considered to be "neural" signals
keyboard movement
blinking facial movements
EKGheart rate, heart rate variabilityhigh SNR, very easy to detect; circuits need to be very safe

II. Signal recording

The devices that record human neural come in many forms, even though they may operate by identical principles. Don’t be confused by the many forms that they purport to take: instead, examine their design and execution by the physical usability of setting up the recording process, the stability and reliability of the resulting recording, and the data value in terms of their spatial and temporal resolution to provide you a useful signal.

Ignore the marketing

Some of them, perhaps the most familiar of which, are marketed as life-style or performance quantification, promising to measure some aspects of your exercise or mood. Some market themselves as toys or presentation aids, allow you to perform “telekinesis” or virtual control. Some actually purport to be EEG, EMG (electromyography), or EOG recording and research tools. Some still do claim to record EEG, but tailor its use specially for educational and demonstration purposes. Whatever their marketed purpose, their significance to you, the neuroengineer, primarily rides upon their design - both in their circuit design and their human-centered usability - to give you the signal features you wish to use.

Recording system comparison

Recording hardwareCommon useUsabilityStabilityData valueLimitations
OpenBCI 32bit + UltracortexP300, α, μ, β, δ, VEPsModerate-High: Fast set-up, many API choices (some in early development), very flexibleModerate: somewhat motion-resistant, good wireless stabilityHigh: adjustable spatial resolution, high sampling rate, modifiable with 3D printerExpensive: 8 channel resolution for $800, and 16 channel can be 1100 or more
OpenBCI Ganglion + UltracortexP300, α, μ, β, δ, VEPsModerate-High: Fast set-up, many API choices (some in early development), very flexibleModerate: somewhat motion-resistant, good wireless stabilityModerate: adjustable spatial resolution, high sampling rate, much more limited channel countModerately expensive: 4 channel resolution for $400
OpenBCI Ganglion + own physical hardwareP300, α, μ, β, δ, VEPs, EMG, EKGModerate: *many* API choices (some in early development), very flexible, but need to design and manufacture make own wearable interfaceModerate: somewhat motion-resistant, good wireless stability. High if used for EMG/EKG.Moderate: adjustable spatial resolution, high sampling rate, much more limited channel count. High if used for EMG/EKG where less channels are needed.Difficult to interface with neural signals with stability; may be great option for EMG or EKG.
Neurosky Mindwaveα/βModerate: Quick to achieve a feed of very limited metricsLow-moderate: motion-susceptible, variable between individualsLow: single-channel low bandwidth recordings; neural onlyInexpensive but very limited data
Emotiv EPOCP300, α, β, μ, VEPsLow-moderate: Can achieve a feed of very limited metrics, Low-moderate: motion-susceptible, variable between individualsModerate: 16-channel recordings; neural onlyLimited data usage, pre-set algorithm outputs, EEG access available only on additional subscription, limited APIs
Emotiv Insightα, βModerate: Quick to achieve a feed of very limited metricsLow-moderate: motion-susceptible, variable between individualsLow: 4-channel low bandwidth recordings; neural only; does not allow raw EEG data outputLimited data usage, pre-set algorithm outputs, no raw EEG access
Thync-----
Versus-----
BitalinoP300, α, β, μ, VEPs, EMG, EKGLow-Moderate: provides circuitry only, wireless; need own streaming and processing stackModerate: when used for EMG/EKG, dependent on your wearable interfacing setupModerate: various channels and individual preamps, can record at high bitrates depending on own circuitryVery versatile and small tools, easy to sew into wearables, but need to do a lot of ground work
FlexVolt SensorEMG, EKGModerate: provides circuitry, wires and electrodes, and some processing stack, wireless; limited API and existing codebaseModerate: when use for EMG/EKG, but there are a lot of excess wiresModerate: 2-8 channel variants at 256 HzLimited existing codebase, need a bit of reverse engineering from existing manufacturer's OS code, may want to replace existing wiring
Backyard Brains Muscle SpikerEMGHigh: 1 channel but works out of box; a quick demoHigh: when used for EMG, existing hardware filters very stableLow-moderate: can obtain pre-filtered and post-filtered (hardware filter) signal, trade off stability with usabilityVery easy to get started and produce a quick demo, but about the same as a very-low-channel (1 ch) bitalino when you want more complex behavior
Backyard Brains Heart & Brain SpikerEEG, EKGHigh: 1 channel but works out of box; a quick demoHigh--moderate: when used for EEG, existing hardware filters very stable, but headband designs less soLow-moderate: can obtain pre-filtered and post-filtered (hardware filter) signal, trade off stability with usability. May need gel use.Very easy to get started, has some codebase and ideas, but about the same as a very-low-channel (1 ch) bitalino when you want more complex behavior

III. Signal processing overview

This is the “glue” between the raw signals and some proxy of usable behavior, and you may need to produce a lot of glue code for differing time resolutions and wearable modalities, especially when creating a hierarchical device that are dependent on many inputs. A large number of common techniques have been developed in the past several decades, with increasing computational complexity, and lagging up to several decades behind cutting-edge compute and learning techniques.

Underdetermined & underconstrained

While at first glance algorithm design may seem to be a good way to make breakthroughs, the dependence of cutting-edge learning algorithms on the stability and SNR of the signal capture cannot be emphasized enough. Overfitting to noise and artifacts of particular datasets is extremely easy due to the lack of SNR of useful features, and because neural behavior is highly underconstrained and underdetermined.

“The brain has about 1014 synapses and we only live for about 109 seconds. So we have a lot more parameters than data. This motivates the idea that we must do a lot of unsupervised learning since the perceptual input (including proprioception) is the only place we can get 10^5 dimensions of constraint per second.” - Geoffrey Hinton

Comparison of feature extraction systems

Preprocessing, noise rejection, spatial filtering

PreprocessingCommon useAdvantageLimitations
Common average referencing (CAR)Common noise rejection, power rejection, motion artifact rejectionVery simple to implement, low compute and memory requirements, $${\text{sig}_i}'=\text{sig}_i-\frac{1}{n}\sum\limits_{i}^{n}\text{sig}_i$$Noise ingress/flux must be similar in magnitude; sensors must be facing the same direction in space. Performs better for power rejection than for motion artifacts. Smears signals spatially. Can destroy out-of-phase traveling waves.
Surface Laplacian, minimal/fixed formCommon noise rejectionWhen implemented as $$\begin{bmatrix} 0&-\frac{1}{4}&0\\ -\frac{1}{4}&1&-\frac{1}{4}\\ 0&-\frac{1}{4}&0 \end{bmatrix}$$ or larger forms, works across curved electrode grids (such as EEG). Lower spatial smearing than CAR. Still very simple to implement.Localized strong noise much more likely to smear over legitimate signals. Very simplistic and prone to bad assumptions.
Surface Laplacian, spherical spline formCommon noise rejection$$\nabla_{\text{surf}}^{2} P_{n}(\cos \theta)=-\frac{n(n+1)}{r^2}P_n(\cos \theta)$$, works across curved EEG grids. Lower spatial smearing than CAR. Better assumptions than above minimal form.Much more annoying to implement than static forms. Requires known geometry or physical model of headform. Slow.
Common spatial pattern (CSP)Pre-extract features specific to block-design contrast conditionsImplemented as $$\mathbf{w}=\text{argmax}_{\mathbf{w}}\frac{||\mathbf{w}\mathbf{X_1} ||^2}{||\mathbf{w}\mathbf{X_2}||^2}$$ gives precomputed spatial weight w for areas with maximum variance between conditions. Signal can be very usable and strong after processing.Requires the signal conditions to be stationary across time; requires a designated and planned training phase with separate recording blocks to obtain w, which can be unreliable.
Principal component analysis (PCA)Find spatial weights that maximize variance across all timeseries (cannot pre-define blocks). Noise rejection, motion artifact rejection. Improve classification.Fast, easy to apply. Implemented as $$\mathbf{X^TX}=\mathbf{W\Sigma^2W^T}$$ Commonly used, requires a training phase but does not need to be pre-planned well. Top PCs often useful for something. Can be used to reject noise (by reprojecting and excluding smallest PCs).Top PCs may not mean anything, and can be a mishmash of multiple phenomena. Single phenomena can be split over several PCs into + and - components. If SNR is low, noise may still dominate variance and therefore PCs. Cannot predefine blocks.
Independent component analysis (ICA)Noise rejection, motion artifact rejection. Improve classification.Excellent at power noise rejection and motion artifact rejection. Generally separates well non-physiological from physiological signals.Very slow compared to all above techniques. Mostly only usable offline. Quite often need an additional stack to identify useful v. non-useful components; this is often done only by eye.

Oscillatory and spectral density

Signal processingCommon useAdvantageLimitations
Gabor filter / short-time (windowed) FFTOscillations, spectral power density, spectral power ratiosExtremely common, size of codebase, fast to implement, often even have hardware instructionsfeature extraction highly sensitive to Gabor frame size
Multitaper FFTOscillationsPrecision determination of oscillatory frequencySlow, latency and low temporal resolution after implementation; needs 3 full oscillations for accurate determination
Wavelet transformsOscillations, spectral power densityRelatively precise determination of oscillatory frequency; approaches the Gabor-Heisenberg theoretical limit; faster than FFT if only need a few spectral features at a timeMuch slower if need as many spectral features as FFT
AR modelsSpectral power densitySimilar features extracted as FFT, but depending on noise level not interchangable with FFT spectral components. Can be faster depending on order.Not as well as documented in neural usage as FFT, except in some cases. Not well-suited for oscillations.
Analytic signal envelopeSpectral power density, instantaneous signal phaseFast to compute for known spectral bands, even for wide bandwidths. Simple to compute acausally as the absolute value of the hilbert analytic transform; phase can be extracted precisely as the cosine angle. More difficult as a causal filter, but can be approximated as FIR.Must be approximated as FIR as a causal filter. Not efficient to use if need many bands, but may be the best way to extract phase.