Breaking out the ol’ EEG headset
Typing with the MindWave Mobile
Some time in college (Go Flames!), I ended up purchasing a MindWave Mobile headset. My plan was to replace typing with a keyboard with some analysis of the waves from your brain. This technology already exists in several forms, but why not try reinventing the wheel!?
I figured that controlling something with your brain would be difficult the first hundred times. I liken this to riding a bike or writing with a pencil. The dream was that humans would have a “mind control” class in grade/middle school just like a typing class. Anyway, I really don’t know anything about the human brain or development but one can dream…
The Beginning
The first step is to find out what data I can get out of the MindWave. Fortunately, there is a GoBot example of interacting with the headset. From the looks of the code snippet, I can get a bunch of data out of the device! However, I soon found that most of the events only fire at about once or twice per minute. Even after finding this, I still decided to build a initial prototype with the following code snippet:
width, _, err := terminal.GetSize(0)
widthF := float64(width/2)
bg := color.New(color.BgYellow)neuro.On(neuro.Event("attention"), func(data interface{}) {
percentAttentive := float64(data.(uint8)) / 100.0
barLength := widthF * percentAttentive
bg.Println(strings.Repeat(" ", int(barLength)))
})
The snippet prints a yellow bar that represents how focused you are. Pretty neat stuff, except that the frequency of events is hardly encouraging. As the yellow bar shrunk, I decided that another solution was necessary.
With research phase starting again, I decided to reevaluate the EEG landscape. This article does a pretty good job at some of the alternative devices. I took note of the OpenBCI project as I figured it would have some inspiration from the DIY community. I watched most of this YouTube video by Conor Russomanno. Conor mentioned that although the thought of reading brain waves is seductive (I think he said the word “sex-appeal” somewhere in there…), EMG (electromyogram) artifacts provide a pretty decent starting place when building a human-computer interactive system. I decided that limiting my scope to electricity emitted from muscle movements (specifically, movements from your jaw) would likely be easy enough to prototype.
The next step was to find an event that represents jaw clenching that fires more that two times per second. The ThinkGear communication protocol (used by MindWave Mobile) emits an “8BIT_RAW Wave Value” event. In our GoBot script, this corresponds to neuro.Event("wave").
With the following code (using this moving average library)
var (
sum int
duration = 100 * time.Millisecond
)// ...a := ewma.NewMovingAverage(100)
ticker := time.NewTicker(duration)
go func() {
for range ticker.C {
fmt.Printf("~~~~~~~~~~~~ %d operations per %s ::: %v\n",
sum,
duration.String(),
a.Value()
)
sum = 0
}
}()
work := func() {
neuro.On(neuro.Event("wave"), func(data interface{}) {
sum = sum + 1
var x int16
x = data.(int16)
a.Add(float64(x))
})
}
I generated the following logs:
~~~~~~~~~~~~ 30 operations per 100ms ::: 36.531944965089814
~~~~~~~~~~~~ 80 operations per 100ms ::: 38.199314524075916
~~~~~~~~~~~~ 40 operations per 100ms ::: 33.82576891258158
~~~~~~~~~~~~ 60 operations per 100ms ::: 70.55005119907753
~~~~~~~~~~~~ 40 operations per 100ms ::: -11.16855924604569
~~~~~~~~~~~~ 60 operations per 100ms ::: 29.164322359335188
~~~~~~~~~~~~ 50 operations per 100ms ::: 56.94057114439317
~~~~~~~~~~~~ 57 operations per 100ms ::: 53.44083002475951
~~~~~~~~~~~~ 49 operations per 100ms ::: 37.95868239142996
~~~~~~~~~~~~ 50 operations per 100ms ::: 36.85669707402736
~~~~~~~~~~~~ 40 operations per 100ms ::: 32.2612409052118
~~~~~~~~~~~~ 60 operations per 100ms ::: 25.745069385336528
~~~~~~~~~~~~ 50 operations per 100ms ::: 57.088639991441205
~~~~~~~~~~~~ 60 operations per 100ms ::: 33.209199281024205
~~~~~~~~~~~~ 40 operations per 100ms ::: 27.49054075271816
~~~~~~~~~~~~ 60 operations per 100ms ::: 33.668075771382505
~~~~~~~~~~~~ 40 operations per 100ms ::: 60.86708042857103
~~~~~~~~~~~~ 60 operations per 100ms ::: -61.28198124948076
~~~~~~~~~~~~ 55 operations per 100ms ::: 48.40010982574792As you can see, the “resting” wave value is around 20/30 but can go as high as 50 or 70. When I clench my jaw, I get EWMA values in the negatives. (-11, -61). However, when I keep my jaw shut, the EWMA value does not stay negative. Also, notice how we are getting around 50/60 events per 100ms.
Like the regular signed 16-bit RAW Wave Value, the
8BIT_RAWWave Value is typically output 128 times a second, or approximately once every 7.8 ms.
It seems that we are getting events at 500 times a second! wow! After a quick typing test, I averaged around 50 words-per-minute and 260 characters-per-minute. 260 / 60 = 4.4 characters per second . Though we haven’t decided what input looks like for my jaw, I think this means that my jaw would need to performs at least 4 clenches per second no matter what algorithm we use. I am going to guess that 4 clenches per second is unreasonable.
Limiting Scope
I use several tools to increase productivity in my day to day tasks. I use the kwm window manager to organize windows on my laptop. I use Karabiner to swap the left Control key with the left Command key. Finally, I use TotalSpaces to disable Desktop animations. The combination of these three means extremely quick switching between workspaces.
Ergonomically, Desktops 1,2,3,4, and 5 are the easiest to switch to. I usually keep Chrome on Desktop 1. Desktops 2,3,4 represent development, usually ranked in importance. In the picture above, I have Intellij open on Desktop 2. Desktop 5 is where I keep Email and HipChat. Other chat applications or overflow windows (like Jabber, Slack, WebEx) end up on Workspaces 6/7. Workspace 9 is reserved for Spotify ❤.
Switching Workspaces with a Jaw Clench
Based on my above research, I have to use a jaw clench as a binary event. Some ideas relating to binary include ASCII (an encoding of binary data) and Morse Code (in that we can use multiple events over time to describe a word). Another useful concept is activation. Think of using “Ok Google” or one of Echo’s wake words to start a verbal transaction. Similarly, we will have to use jaw clenches to start a clench transaction.
Starting small, I’ll build a event that fires when two jaw clenches happen in quick succession.
First, we need a function to be called when a clench happens.
func ClenchListener(clenchChan chan bool) {
for {
clench := <- clenchChan
if clench {
fmt.Println("Clenched")
}
}
}Next, we need to trigger the clench event.
clenches := make(chan bool)
go ClenchListener(clenches)for range ticker.C {
sum = 0
clench := a.Value() < clenchThreshold
clenches <- clench
}
Some issues I faced were that subsequent clenches didn’t trigger a clench event. I think this was because the weighted moving average was over too many events and couldn’t reach “resting” state between events. Another issue I found was that voluntary blinking and jaw clenching resulted in clench events. This resulted in the following refactored code:
emgEvent := <-emgChan
if emgEvent.RawValue.Value < emgThresholdMax && emgEvent.RawValue.Value > emgThresholdMin {
if IsReflexBlinkEvent(emgEvent.RawValue) {
fmt.Println(EMGEventTypeReflexBlink)
} else if IsVoluntaryBlinkEvent(emgEvent.RawValue) {
fmt.Println(EMGEventTypeVoluntaryBlink)
} else if IsClenchEvent(emgEvent.RawValue) {
fmt.Println(EMGEventTypeClench)
}
}And the following logs (with me jaw clenching at a constant rate):
~~~~~~~~~~~~ 40 operations per 70ms ::: 51.22058119761758
~~~~~~~~~~~~ 30 operations per 70ms ::: -10.798668200542943
clench
~~~~~~~~~~~~ 40 operations per 70ms ::: 50.535982311534845
~~~~~~~~~~~~ 35 operations per 70ms ::: 55.175282782502144
~~~~~~~~~~~~ 40 operations per 70ms ::: 50.91739779262282
~~~~~~~~~~~~ 40 operations per 70ms ::: 35.248467300991315
~~~~~~~~~~~~ 40 operations per 70ms ::: 14.31102689944405
~~~~~~~~~~~~ 40 operations per 70ms ::: 40.116955085788476
~~~~~~~~~~~~ 20 operations per 70ms ::: 5.086195366946679
rblink
~~~~~~~~~~~~ 30 operations per 70ms ::: 40.78507387156811
~~~~~~~~~~~~ 40 operations per 70ms ::: 29.768259339461874
~~~~~~~~~~~~ 40 operations per 70ms ::: 36.81051877348129
~~~~~~~~~~~~ 40 operations per 70ms ::: 33.082669643094206
~~~~~~~~~~~~ 30 operations per 70ms ::: -36.49279751026391
clench
~~~~~~~~~~~~ 40 operations per 70ms ::: 60.42703202453221
~~~~~~~~~~~~ 40 operations per 70ms ::: 81.09601717620939
~~~~~~~~~~~~ 40 operations per 70ms ::: 43.45056113074797
~~~~~~~~~~~~ 36 operations per 70ms ::: 12.353704469786463
~~~~~~~~~~~~ 20 operations per 70ms ::: 11.915685096987692
~~~~~~~~~~~~ 40 operations per 70ms ::: 64.33775516695265
~~~~~~~~~~~~ 50 operations per 70ms ::: 0.6421291720932084
rblink
~~~~~~~~~~~~ 30 operations per 70ms ::: -22.357272075254215
clench
~~~~~~~~~~~~ 40 operations per 70ms ::: 46.199103154523485
~~~~~~~~~~~~ 30 operations per 70ms ::: 54.550135015728614
~~~~~~~~~~~~ 30 operations per 70ms ::: 40.67282881702697
~~~~~~~~~~~~ 40 operations per 70ms ::: 26.57810001487704
~~~~~~~~~~~~ 20 operations per 70ms ::: 25.02495127105816
~~~~~~~~~~~~ 50 operations per 70ms ::: 32.41563046109884
~~~~~~~~~~~~ 40 operations per 70ms ::: 20.557655500294192
~~~~~~~~~~~~ 30 operations per 70ms ::: 24.44074781653556
~~~~~~~~~~~~ 40 operations per 70ms ::: 19.484044453787583
~~~~~~~~~~~~ 29 operations per 70ms ::: 37.672988833256596
~~~~~~~~~~~~ 36 operations per 70ms ::: 42.513570658716475
~~~~~~~~~~~~ 40 operations per 70ms ::: 70.36934077726973
~~~~~~~~~~~~ 40 operations per 70ms ::: -24.47829647327087
clench
~~~~~~~~~~~~ 40 operations per 70ms ::: 12.83733399918889
~~~~~~~~~~~~ 40 operations per 70ms ::: 72.98075610980655
~~~~~~~~~~~~ 30 operations per 70ms ::: 70.91724728359634
~~~~~~~~~~~~ 30 operations per 70ms ::: 45.773079699234714
~~~~~~~~~~~~ 20 operations per 70ms ::: 31.756782183696174
~~~~~~~~~~~~ 50 operations per 70ms ::: 4.80399385323248
rblink
~~~~~~~~~~~~ 40 operations per 70ms ::: 58.4482715284451
~~~~~~~~~~~~ 40 operations per 70ms ::: 61.01019345030721Hard-Coded EMG-EMWA ranges got pretty close, but still produced quite a few errors. Here is the range for the above sample:
emgEventTypeReflexBlinkThresholdMax = 10.0
emgEventTypeReflexBlinkThresholdMin = -10.0
emgEventTypeClenchThresholdMax = -10.0
emgEventTypeClenchThresholdMin = -70.0
emgEventTypeVoluntaryBlinkThresholdMax = -70.0
emgEventTypeVoluntaryBlinkThresholdMin = -200.0At this point, the whole experiment feels pretty messy. Actions like squinting or blinking to make my eyes less dry aren’t all that uncommon. I have some suspicion that there is a better interface to measuring jaw clenches that would ignore those movements.
Adding the Double Clench Event
Let’s define a “Double Clench Event” as two clench events that happen at least 250 milliseconds apart but no more than 750 millisecond apart.
emgSynthesizedEventDoubleClenchSpacingMax = 700 * time.Millisecond
emgSynthesizedEventDoubleClenchSpacingMin = 250 * time.MillisecondFirst, we need a place to store a series of events.
import (
lane "gopkg.in/oleiade/lane.v1"
)type EMGEventHistory struct {
History *lane.Deque
EventSpacing time.Duration
}
Now, we need some code to compare the events stored in the History Deque.
func IsDoubleClenchEvent(h *EMGEventHistory) (bool) {
foundDoubleClench := false
if h.History.First().(EMGEvent).Type == EMGEventTypeClench {
spacing := 0 * time.Millisecond
for i := 0; i < h.History.Size(); i++ {
el := h.History.Shift().(EMGEvent)
// if another event is clench and we are in the range
if el.Type == EMGEventTypeClench
&& spacing > emgSynthesizedEventDoubleClenchSpacingMin
&& spacing < emgSynthesizedEventDoubleClenchSpacingMax
{
foundDoubleClench = true
}
spacing = spacing + h.EventSpacing
h.History.Append(el)
} }
return foundDoubleClench
}
This should be enough code to place in our control loop. I’ve added the notifize package to get events to show up.
if IsDoubleClenchEvent(history) {
fmt.Println(EMGSynthesizedEventDoubleClench)
history.Normalize()
notifize.Display("Double Clench", "Clench 1", false, "")
}Reacting to the Double Clench Event
Now that we have a code path that executes when a double clench event comes our way, we need some business logic to count clench events following the double clench.
shouldTriggerEvents = true
shouldStopTriggerTimer := time.NewTimer(doubleClenchTimeout)
go func() {
<-shouldStopTriggerTimer.C
shouldTriggerEvents = false
clenchCount = 0
}()
clenchCount = 1And a single clench event handler:
} else if IsClenchEvent(emgEvent.RawValue) {
emgEvent.Type = EMGEventTypeClench
fmt.Println(EMGEventTypeClench)
if shouldTriggerEvents {
clenchCount = clenchCount + 1
notifize.Display("Clench", fmt.Sprintf("%d", clenchCount), false, "")
}
}This should result in notifications for subsequent clenches.
Doing Something Useful
The last step is to do something useful: Switch Workspaces.
As detailed above, we will need to hit Control + <number key> to move to a workspace. On a Mac, AppleScript is the easiest way (really, a hack) to get something working.
After some trial and error in the AppleScript Editor, I ended up with the following script:
tell application "System Events"
key code 18 using control down
end tellThis link has a great picture diagramming the key code to key mapping.
We are left with the Go function:
func ExecApplescript(scriptNum int) {
OSASCRIPT := "osascript"
p := path.Join(applescript_directory, fmt.Sprintf("workspace_%d.applescript", scriptNum))
fmt.Printf("Execing %s", p)
cmd := exec.Command(OSASCRIPT, p)
err := cmd.Start()
if err != nil {
fmt.Printf("%s")
}
}Final Result
While far from perfect, the code is at a point where it works 40–60% of the time. I used OBS for capture, Snagit for cropping the video, and this ffmpeg script for gif creation.
The above project took around 10 hours of development over 2 weekends. Code is located here:
