Enactive Interface Perception and Affordances

Posted in artificial intelligence, interfaces, philosophy on November 14th, 2011 by Samuel Kenyon

There are two freaky theories of perception which are very interesting to me not just for artificial intelligence, but also from a point of view of interfaces, interactions, and affordances. The first one is Alva Noë’s enactive approach to perception. The second one is Donald D. Hoffman’s interface theory of perception.

Enactive Perception vs. Interface Perception

Enactive Perception

The key element of the enactive approach to perception is that sensorimotor knowledge and skills are a required part of perception [1].

In the case of vision, there is a tradition of keeping vision separate from the other senses and sensorimotor abilities, and also as treating it as a reconstruction program (inverse optics). The enactive approach suggests that visual perception is not simply a transformation of 2D pictures into a 3D representation, and that vision is dependent on sensorimotor skills. Indeed, the enactive approach claims that all perceptual representation is dependent on sensorimotor skills.

Example of optical flow (one of the ways to get structure from motion)

My interpretation of the enactive approach proposes that perception co-evolved with motor skills such as how our bodies move and how our sensors, for instance, eyes, move. A static 2D image can not tell you what color blobs are objects and what are merely artifacts of the sensor or environment (e.g. light effects). But if you walk around this scene, and take into account how you are moving, you get a lot more data to figure out what is stable and what is not. We have evolved to have constant motion in our eyes via saccades, so even without walking around or moving our heads, we are getting this motion data for our visual perception system.

Of course, there are some major issues that need to be resolved, at least in my mind, about enactive perception (and related theories). As Aaron Sloman has pointed out repeatedly, we need to fix or remove dependence on symbol grounding. Do all concepts, even abstract ones, exist in a mental skyscraper built on a foundation of sensorimotor concepts? I won’t get into that here, but I will hopefully return to it in a later blog post.

The enactive approach says that you should be careful about making assumptions that perception (and consciousness) can be isolated on one side of an arbitrary interface. For instance, it may not be alright to study perception–or consciousness–by looking just at the brain. It may be necessary to include much more of the mind-environment system–a system which is not limited to one side of the arbitrary interface of the skull.

Perception as a User Interface

human-computer interfaces (this still from Matrix Reloaded)

human-computer interfaces

The Interface Theory of Perception says that “our perceptions constitute a species-specific user interface that guides behavior in a niche” [2].

Evolution has provided us with icons and widgets to hide the true complexity of reality. This reality user interface allows organisms to survive better in particular environments, hence the selection for it.

Perception as an interface

Or as Hoffman et al summarize [3] the conceptual link from computer interfaces:

An interface promotes efficient interaction with the computer by hiding its structural and causal complexity, i.e., by hiding the truth. As a strategy for perception, an interface can dramatically trim the requirements for information and its concomitant costs in time and energy, thus leading to greater fitness. But the key advantage of an interface strategy is that it is not required to model aspects of objective reality; as a result it has more flexibility to model utility, and utility is all that matters in evolution.

Besides supporting the theory with simulations, Hoffman [2] uses a colorful real world example: he describes how male jewel beetles use a reality user interface to find females. This perceptual interface is composed of simple rules involving the color and shininess of female wing cases. Unfortunately, it evolved for a niche which could not have predicted the trash dropped by humans that lead to false positives. This results in male jewel beetles humping empty beer bottles.

Male Australian jewel beetle attempting to mate with a discarded “stubby” (beer bottle)

For more info on the beetles, see this short biological review [4] which includes “discussion regarding the habit of the males of this species to attempt mating with brown beer-bottles.” It also notes:

Schlaepfer et al. (2002) point out that organisms often rely on environmental cues to make behavioural and life-history decisions. However, in environments which have been altered suddenly by humans, formerly reliable cues might no longer be associated with adaptive outcomes. In such cases, organisms can become trapped by their evolutionary responses to the cues and experience reduced survival or reproduction (Schlaepfer et al., 2002).

All perception, including of humans, evolved for adaptation to niches. There is no reason or evidence to suspect that our reality interfaces provide “faithful depictions” of the objective world. Fitness trumps truth. Hoffman says that Noë supports a version of faithful depiction within enactive perception, although I don’t see how that is necessary for enactive perception.


One might think of perception as interactions within a system. This system contains the blobs of matter we typically refer to as an “organism” and its “environment.”

You’ll notice that in the diagram in the previous section, “environment” and “organism” are in separate boxes. But that can be very misleading. Really the organism is part of the environment:

Of course, the organism itself is part of the environment.

True Perception is Right Out the Window

How do we know what we know about reality? There seems to be a consistency at our macroscopic scale of operation. One consistency is due to natural genetic programs–and programs they in turn cause–which result in humans having shared knowledge bases and shared kinds of experience. If you’ve ever not been on the same page as somebody before, then you can imagine how it would be like if we didn’t have anything in common conceptually. Communication would be very difficult. For every other entity you want to communicate with, you’d have to establish communication interfaces, translators, interpreters, etc. And how would you even know who to communicate with in the first place? Maybe you wouldn’t have even evolved communication.

So humans (and probably many other related animals) have experiences and concepts that are similar enough that we can communicate with each other via speech, writing, physical contact, gestures, art, etc.

But for all that shared experience and ability to generate interfaces, we have no inkling of reality.

Since the interface theory of perception says that our perception is not necessarily realistic, and is most likely not even close to being realistic, does this conflict with the enactive theory?

Noë chants the mantra that the world makes itself available to us (echoing some of the 1980s/1990s era Rod Brooks / behavioral robotics approach of “world as its own model”). If representation is distributed in a human-environment system, does it have to be a veridical (truthful) representation? No. I don’t see why that has to be the case. So it seems that the non-veridical nature of perception should not prevent us from combining these two theories.


A chair affords sitting, a book affords turning pages.

A chair affords sitting, a book affords turning pages.

Another link that might assist synthesizing these two theories is that of J.J. Gibson’s affordances. Affordances are “actionable properties between the world and an actor (a person or animal)” [5].

The connection of affordances to the enactive approach is provided by Noë (here he’s using an example of flatness):

To see something is flat is precisely to see it as giving rise to certain possibilities of sensorimotor contingency…Gibson’s theory, and this is plausible, is that we don’t see the flatness and then interpret it as suitable for climbing upon. To see it as flat is to see it as making available possibilities for movement. To see it as flat is to see it, directly, as affording certain possibilities.

Noë also states that there is a sense in which all objects of perception are affordances. I think this implies that if there is no affordance relationship between you and a particular part of the environment, then you will not perceive that part. It doesn’t exist to you.

The concept of affordances is also used, in a modified form, for interaction design as well. For those who are designers or understand design, you can perhaps understand how affordances in nature have to be perceived by animals so that they can survive. It is perhaps the inverse of the design problem–instead of making the artifact afford action for the user, the animal had to make itself comprehend certain affordances through evo-devo.

Design writer Don Norman makes the point to distinguish between “real” and “perceived” affordances[5]. That makes sense in the context of his examples such as human-computer interfaces. But are any affordances actually real? And that gets back into the perception as interface theory–animals perceive affordances, but there’s no guarantee those affordances are veridical.

1. Noë, A., Action in Perception, Cambridge, MA: MIT Press, 2004.
2. Hoffman, D.D., “The interface theory of perception: Natural selection drives true perception to swift extinction” in Dickinson, S., Leonardis, A., Schiele, B.,&Tarr, M.J. (Eds.), Object categorization: Computer and human vision perspectives. Cambridge, UK: Cambridge University Press, 2009, pp.148-166. PDF.
3. Mark, J.T., Marion, B.B.,&Hoffman, D.D., “Natural selection and veridical perceptions,” Journal of Theoretical Biology, no. 266, 2010, pp.504-515. PDF.
4. Hawkeswood, T., “Review of the biology and host-plants of the Australian jewel beetle Julodimorpha bakewelli,” Calodema, vol. 3, 2005. PDF.
5. Norman, D., “Affordances and Design.” http://www.jnd.org/dn.mss/affordances_and_design.html

Image credits: iamwilliam, T. Hawkeswood [4], Matrix Reloaded (film), Old Book Illustrations.
Diagrams created by Samuel H. Kenyon.

This is an improved/expanded version of an essay I originally posted February 24th, 2010, on my blog SynapticNulship.

Tags: , , , , , , ,

GUI Prototyping at a BostonCHI Seminar

Posted in interaction design on March 28th, 2011 by Samuel Kenyon

On Friday I attended BostonCHI’s seminar Tools of the Trade: User Experience Research and Design Skills. Since all courses were day-long, I had to choose only one.  My choice was Prototyping Tips and Tools for Effective UX Design.

Here is an embedding of the instructor’s Prezi used during the course.  It’s pretty good, except for the right-brain/left-brain crap, which is a myth.

A few things I learned:

  • (meta) Prezi seems to be an extremely useful tool for presentations and/or videos
  • CaseComplete appears to be a very useful tool for managing use cases and requirements and traceability between everything, even to test plans.
  • FlairBuilder is a pretty good tool for prototyping. You can also use it for wireframing.  Unfortunately, it still has some major bugs (it crashed several times for everybody in the class).  The file format use XML; it’s simple enough to read it manually and I messed around with a bit and reloaded that modified file (the program didn’t choke with my hacked files, even when I purposely did weird things).  Anyway, I like the fact that somebody else could easily write a program or script to reuse one’s Flair files, for instance creating visualizations of how all the elements are connected or activity diagrams.
  • FlairBuilder card stacks are very useful.  For some, it’s a new concept; for me, I had fond memories of card stack apps I’ve used in the distant past such as HyperStudio and one I made myself in high school with QuickBasic.
  • Prototyping tools like FlairBuilder and Axure are worth using if you need to demonstrate a GUI with lots of transitions, etc. that would take a long time to actually code.

Prototyping may be more associated with web design but I have found it to be useful for other kinds of GUIs.  Indeed, I am not a web designer at all.  But I have no problem stealing good ideas from web design.  In my experience, a working demo is better, but if you can’t do that in time (or if it would be a waste of effort) then prototype or at least make static mockups.

Tags: , , , ,

Fiber Optic Neural Interfaces: Tests to Begin Soon

Posted in interfaces, transhumanism on March 2nd, 2011 by Samuel Kenyon

Popular Science [1] has reported a tidbit of information: Marc Christensen’s team at SMU is supposed to start testing if they can stimulate a rat’s leg with optical fibers.

Fiber optic to nervous system interface

Fiber optic to nervous system interface

This is the same DARPA-funded project I mentioned last September in my article “Softer, Better, Faster, Stronger” [2].  DARPA held a related “Reliable Neural Interface Technology (RE-NET)” workshop back in 2009 [3]:

A well-meaning motor prosthesis with even 90% reliability, such as a prosthetic leg that fails once every 10 steps, would quickly be traded for a less capable but more reliable alternative (e.g., a wheelchair). The functionality of any viable prostheses using recorded neural signals must be maintained while the patient is engaged in or has their attention directed to unrelated activities (e.g., moving, talking, eating, etc.). Since the neural-prosthesis-research community has yet to demonstrate the control of even a simple 1-bit switch with a long-term high level of speed and reliability, the success of more ambitious goals (e.g., artificial limbs) are placed in doubt.

DARPA is interested in identifying the specific fundamental challenges preventing clinical deployment of Reliable Neural Technology (RE-NET), where new agency funding might be able to advance neural-interface technology, thus facilitating its great potential to enhance the recovery of our injured servicemembers and assist them in returning to active duty.


Technology comparison

Some of the challenges listed for the optical (neurophotonic sensing) approach are [4][5]:

  • Transduce action potential into optically measurable quantity
  • Modes: ionic concentration / flux vs. electromagnetic field
  • Field Overlap
  • Can’t go straight from voltage (indirect detection)
  • Sensitivity, Parallelism
  • Packaging, Size
  • Untested
  • “What is the minimum level of control-signal information required to recover a range of activities of daily living in both military and civilian situations?”
  • “Need a method for characterizing tissue near implant to better understand long term degradation.”

Some of those challenges probably apply to all forms of neuro sensing.  Likewise, the metrics for neurophotonic interfaces—resolution, signal-to-noise ratio, and density—probably apply to other methods as well.

The Need for Better Neural Interfaces

Future prosthetics

Future prosthetics

Maybe the neurophotonic approach won’t work in the end, or it will only work in combination with another method.  Whatever the case, a lot of money should be put into this kind of project.  We are in desperate need for more advanced neural interfaces.  As Dr. Principe of the University of Florida writes [6]:

Just Picture yourself being blindfolded in a noisy and cluttered night club that you need to navigate by receiving a voice command once a second…And you will understand the problem faced by engineers designing a BMI [Brain Machine Interface].

Present systems are signal translators and will not be the blue print for clinical applications.  Current decoding methods use kinematic training signals – not available in the paralyzed. I/O models cannot contend with new environments without retraining.  BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user.

Interfaces to the nervous systems are the key enablers for all of future prosthetics—and of course other exotic devices that don’t even exist yet.  Without overcoming this interface hurdle, we’ll be stuck in the stone age of prosthetics and nervous system repair.

[1] M. Peck, “Talk To The Hand: A New Interface For Bionic Limbs,” Popular Science, Feb 24, 2011.
[3] J.W. Judy & M.B. Wolfson, RE-NET website.
[2] “Softer, Better, Faster, Stronger: The Coming of Soft Cybernetics,” H+ Magazine, Sept 21, 2010.
[4] M.P. Christensen, “Neuro-photonic Sensing: Possibilities & Directions”, DARPA RE-NET Workshop, Nov 19, 2009.
[5] Optical Breakout Session Report, DARPA RE-NET Workshop, Nov 20, 2009.
[6] J.C. Principe, “Architectures for Brain-Machine Interfaces,”  DARPA RE-NET Workshop, Nov 19, 2009.

Image Credits:
[1] Rajeev Doshi, PopSci

[2] DARPA / CIPhER via Physorg
[3] scan of book cover, art by John Berkey

Tags: , , ,

Metaphysics of Interfaces

Posted in interfaces, philosophy on December 22nd, 2010 by Samuel Kenyon

We have an everyday sense of interfaces.  The computers we use all have interfaces, both in software and hardware.  If they didn’t, we wouldn’t be able to use them (of course, some interfaces are clearly better than others).  But interfaces aren’t just for computers—every tool or entertainment device has interfaces.  For instance the size and shape of a hammer or a pistol affords a certain usage by human hands which is very effective, and even comfortable.

But is there a more fundamental, general concept of interface?

First, we can enumerate a few of the more important roles that our common human interfaces can take: Interfaces can be thought of as translators, for instance human-computer interfaces translate a machine language into something humans can deal with such as text and/or graphics.  Interfaces can be masks, for instance avatars and augmented reality insert a layer of reality modification between users and worlds.  Interfaces can connect different types of substrates, for instance biological to electronics.  Interfaces can connect objects of different scales, for instance the interfaces of heavy machinery allow a single human to move massive quantities of material (or in a somewhat less common example, a human can manipulate specific atoms with the interfaces provided by a scanning tunneling microscope).

There are other types of interfaces, such as chemical surface boundaries between two phases.  Biology has various kinds of interfaces; computer science has its kinds of interfaces; and so on.  Basically, whenever two or more objects interact, there is an interface at that interaction.  Some interfaces are natural, and some are designed to make the interaction between the objects effective.  But there doesn’t need to be a third thing that is the interface.  The interface can be the transient place at which two or more things intersect.

What is the metaphysical situation for interfaces?  Do interfaces exist as universals?  Are they abstract?  Are they objective or subjective?  Let’s say that I am ontologically committed to the existence of objective interfaces.  So these could be concrete, but can an interface in its simplest form be concrete or must it be abstract?  Perhaps there is a universal interface—a class of which all interfaces are instances of.  This would posit that the phenomenon of interfacing is the same at all scales and regardless of whatever particulars were involved in the interfacing.

Now, let’s say that we thought there were real world instances everywhere of the universal interface.  At what scales would that stop?  Is there some underlying level in which entities no longer interface?

Now, why would I even bother to think about objective abstract interfaces?  Because, it’s possible that interfaces at the simplest conception are the basic connector of objects.  If that premise is true, then without the existence of objective interfaces there would not objectively exist anything separate from anything else—or there could be but they would effectively be in their own universes because they would never be able to interact.

If objective interfaces do not actually exist in this world, then we have to deal with the concept of interface just as a metaphor.

At the human scale, discussing interfaces seems to embrace an object-oriented point of view, which is basically the natural human point of view.  Humans operate largely by perceiving the world in terms of objects, with agents being a special class of object that operate autonomously.  Other humans are agents, other animals are agents, anything that appears to move by its own volition is suspicious and given at the very least temporary status as an agent.  But are objects, i.e. particular entities, necessary for the concept of interface?  Perhaps an objective theory of interface would not require objects.  Maybe objects are just slices of the world which are convenient for our minds to process.  Although it seems like we interface with objects, it’s possible that all interfaces operate between folds of the same cloth—some continuity that is not composed of objects (or the world itself is the only object).

Cross-posted with Science 2.0.

Tags: , ,