EU Project MACS Description Page

home

about

partners

events

MACS in the press

publications

downloads

links

private

	Sitemap Contact Imprint

Project executive summary

Multi-sensory Autonomous Cognitive Systems Interacting with Dynamic Environments for Perceiving and Using Affordances

A Research Project funded by the European Commission in their 6^th Framework Programme under grant no. FP6-004381 within the “Cognitive Systems and Robotics” unit.

1. Final Activity Report
2. Project achievements
    a. Overview
    b. Conceptual work, implementation and integration
    c. Experimentation scenarios
    d. Dissemination
3. Contractors involved (see also partner links)
4. Background: Affordances
5. Progress beyond the state of the art:
    a. Formalisation
    b. Representation for deliberation and planning
    c. Perception
    d. Learning
    e. Architecture
6. References

1. Final Activity Report

Publishable Final Activity Report of MACS

2. Project achievements

In the following, references of type [Dx.y.z] refer to public deliverables of MACS.

2a. Overview:

The main objective of the MACS project was to explore and exploit the concept of affordances for the design and implementation of autonomous mobile robots acting goal-directedly in a dynamic environment. The claim was to develop affordance-based control as a method for robotics. Technically speaking, a prototypical affordance-based architecture (Fig. 4) has been developed and implemented on a mobile robot, KURT3D (Fig. 1). For providing simple manipulation capabilities, KURT3D is equipped with a magnetic gripper crane arm. The experiments have been performed in the simulator MACSim (Fig. 3) and in a real demonstrator scenario (Fig. 2).

Although other researchers occasionally attempted to relate affordances to robotics ([Mur99],[MDor00],[Fit03],[Sto05a],[Sto05b]), it is valid to state that MACS was the first dedicated investigation of using the concept of affordances in all stages of robot control, in perception, learning and planning. MACS has achieved its general goal "to explore and exploit the concept of affordances for the design and implementation of autonomous mobile robots acting goal-directedly in a dynamic environment." On this page, we summarise the MACS results and their relation to the project's objectives.

As far as perception of affordances is concerned, the concept of affordance cueing, i.e. applying a trained 'matched filter' when looking for affordances, is considered to be an important project result. Affordance cueing provides perceptual economy for robot perception, since it uses only a relevant fraction of the available perceptual input. Affordance cueing has been implemented in two different ways, namely first by using reinforcement learning for training, and second by using Support Vector Machines for training classifiers (the 'matched filters').

Although it has not been an initial goal of MACS, it turned out in the first year of the project that we need a formalisation of the affordance notion in order to make use of it. Such a fo-malisation provides clear definitions that aid defining representations and implementing the control architecture. A first formalisation has been drafted in our deliverable [D4.2.1+4.3.1]. It provided first definitions of agent affordances and their representations. Later in the project, a second formalisation has been developed that introduced a formalisation of the equivalence concepts (entity, affordance, and outcome/effect equivalence) that are the basis for the proposed flexibility. This second formalisation has been published in the Adaptive Behavior journal, and thus MACS has also fulfilled its aim to embed its results into Cognitive Science.

Fig. 1: Simple experimentation environment, showing robot KURT3D with 3D Laser scanner. The crane arm includes an electromagnet, enabling the robot to grip and lift magnetic objects (FhG/AIS).
Fig. 2: Demonstrator scenario (FhG/AIS).

Fig. 3: Scene from the ODE-based MACS simulator MACSim (METU-KOVAN).
Fig. 4: MACS affordance-inspired robot control architecture (All). Details can be found in Final Activity Report, D2.2.2, and [Rom08b].

top of page

2b. Conceptual work, implementation and integration:

This activity has produced concrete proposals for the control architecture, learning, perceiving and representing affordances. These are described both in the deliverables (cf. below), and in the project publications. Perception of affordances is feature-based [D3.1.1, D3.1.2, D3.1.4] (Fig. 7) and employs a visual attention system [D3.1.3], (Fig. 6). The proposed multi sensory and sensorimotor based affordance perception is described in [D3.2.1, D3.3.1, D3.3.2]. Learning produces relational descriptions of affordances in a four-stage process [D5.2.1, D5.3.1, D5.3.2, D5.4.2]. An outlook on affordance learning by observation is presented in [D5.4.5]. A formal theory of affordances, a suggestion for representing affordances based on the concept of tolerance spaces, and a first description of an affordance-based motion planner is described in a combined deliverable [D4.2.1+D4.3.1], (Fig. 8). An alternative formalization has been published in Adaptive Behavior. A common affordance inspired robot control system including perception module, behaviour system, execution control module, planner, learning module and affordance representation repository has been implemented, tested and used (MACS architecture, Fig. 2, [D2.2.2]). D2.2.2 was the central document for the implementation of the MACS architecture and gives the most details on the overall architecture. Representation of affordances for deliberation and a planner that makes use of affordance representations (Fig. 5) are described in the deliverables [D2.3.2, D4.3.3, D4.4.3]. Work on all modules of the MACS architecture has been concluded. Perception module, planning module, execution control module, behavior system and user interface have been integrated and demonstrated both in simulation and on the real robot.

top of page

2c. Experimentation scenarios:

The proof-of-concept has been shown in various experiments with the simulator MACSim and the real robot KURT3D. The demonstrator scenarios ([D6.1.1], Fig. 1) and a scheme for designing experiments [D6.4.1] have been specified. A complex stacking demonstrator scenario has been shown as final demonstrator. The MACS simulator MACSim (Fig. 2), developed and optimized by METU-KOVAN [D1.2.1, D6.2.1], has played a central role in experimentation and evaluation. Four KURT3D robots (Fig. 3) were available at OFAI, FhG/AIS, METU-KOVAN and University of Osnabrück [D6.3.1]. They have been used both for preliminary experiments and for the final demonstration.

2d. Dissemination:

The dissemination of project results is a standard goal for all EU research projects. MACS has produced 18 peer-reviewed listed publications, two edited books, five book contributions, four theses, two online articles and twelve posters. Research results have been presented at all major Robotics conferences (IROS, ICRA, ICDL, SAB), other relevant conferences and workshops, and at seminars and invited talks. MACS has organized a Dagstuhl seminar dedicated to its central topic, affordances for robot control. The proceedings appeared as a Springer LNAI volume [Rom08a] and contain an overview article on the MACS approach [Rom08b].

The general public has been addressed by several press releases that have been echoed in print media and on more than two dozens web sites (see here. An Industry Day for informing interested companies about the research being performed in the EU's Cognitive Systems objective has been organized and conducted by MACS, with the support of the Coordination Action euCognition. At the Industry Day, six projects, including MACS, have been presented to 47 attendees, half of them from industry. A detailed list of the project's dissemination events can be found here, a list of publications here. All in all, MACS has achieved its dissemination objective.

Fig. 5: Plan-based execution using affordances. Robot KURT3D putting a test object on the switch for opening the sliding doors (All partners).
Fig. 6: Using simulated visual attention for exploration: tracking test object (FhG/AIS).

Fig. 7: Feature based entity support matrix for describing the distinctive features (cues) that indicate the presence of a liftability affordance (JR_DIB).
Fig. 8: Elevation map, segmented modulo robot maneuvrability. Experiments performed by LiU-IDA.

top of page

3. Contractors involved

The co-ordinator of the MACS project was the Fraunhofer Institute for Intelligent Analysis and Information Systems in Sankt Augustin, Germany (FhG/AIS). FhG/AIS provided expertise in mobile robotics, control architectures, AI, sensor systems and biologically motivated perception. Joanneum Research, from Graz, Austria (JR_DIB), combined know-how in cognitive visual perception systems and in machine learning. Linköpings Universitet, Sweden (LiU-IDA), contributed know-how in AI, knowledge representation and robotics. Middle East Technical University, Ankara, Turkey (METU-KOVAN), provided its know-how in the simulation of robotic systems and in control architectures. Cognitive science and learning methods are the research topics of OFAI, Österreichische Studiengesellschaft für Kybernetik, Vienna, Austria. The University of Osnabrück contributed expertise in AI planning, knowledge-based systems and robotics.

top of page

4. Background: Affordances

Experimental psychologist James J. Gibson coined the notion of affordances in his "Ecological Approach to Visual Perception":
"The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, but the noun affordance is not. I have made it up. I mean by it something that refers both to the environment and the animal in a way that no existing term does. It implies the complimentarity of the animal and the environment." ([Gib79], p. 127)

The animal (or agent), on the other hand, must possess the capabilities to perceive and act upon the affordances. An affordance in this sense is a very specific relation between an agent and an environment that provides a capability or potential for action. The potential is direct in the sense that perception is direct. Perceptual entities are not semantic abstractions but opportunities for action mediated by a specific relation between the agent and its environment. This implies a very tight loop between actions (or behaviors) and opportunities the environment affords (perceptual entities). The behaviors and perceptual entities are symbiotic.

Another important aspect of Gibson's theory of affordances is that "... to perceive an affordance is not to classify an object." ([Gib79], p. 134). Gibson goes on to state that "... If you know what can be done with a graspable object, what it can be used for, you can call it whatever you please. ... The theory of affordances rescues us from the philosophical muddle of assuming fixed classes of objects, each defined by its common features and then given a name. ...But this does not mean you cannot learn how to use things and perceive their uses. You do not have to classify and label things in order to perceive what they afford." ([Gib79], p. 134)

Thus, objects and affordances are complementary in the sense that one object class may offer a multitude of affordances, and one affordance may be offered by a multitude of object classes.

After this brief characterization, we illustrate why the concept of affordances may be beneficial for the use in robot control: Firstly, the concept of affordances links perception, action, learning and reasoning in an agent specific way. Thus it bears the potential for a new engineering method for a biologically inspired, hybrid (reactive and deliberative) control architecture for mobile robots with manipulation capabilities.

Secondly, the complementarity of the object and the affordance notions allow a robot a greater flexibility for performing tasks. A robot system that uses object-centered perception may need to abort a mission if objects of a certain class that are required to reach a (sub-)goal are not available. In those cases where an affordance (like a function) of this object is more important than its sensorial appearance, affordance-based perception may be more appropriate, since it allows the robot to perceive and use objects with the same function that belong to a completely different object class, that is, it helps finding alternatives for action.

In the MACS project, affordances were built into a robot control architecture as 1st-class citizens. There is explicit support for the concept in our architecture. The hypothesis that the resulting performance of the robot will benefit in terms of robustness and generality has been validated. In fact, these are essentially the only criteria that can be used to empirically evaluate whether an affordance-based robotic system is better than a non-affordance-based one.

Although Gibson provides a rich illustration of the concept of affordances in his “Theory of Affordances” [Gib77], he never provided a crisp, let alone a formal definition of affordances. This makes it difficult to use the concept in a technical system. Also, he elaborated much on perceiving affordances, but little on using or learning them. Important tasks of MACS, thus, were conceptual work on formalizing and operationalizing the concept of affordances.

top of page

5. Progress beyond the state of the art

References of type [Dx.y.z] refer to public deliverables of MACS.

5a. Formalisation:

We believe that the proposed formalism [Sah07] has laid out a good framework over which the concept of affordance can be utilized for autonomous robot control. The review of the affordance concept and its use in different fields, with special emphasis on Autonomous Robotics has provided a complete snapshot of the confusion surrounding the concept and through the formalisation has provided its implications at different aspects of robot control ranging from learning to planning. This is clearly a progress beyond state-of-the-art since the affordance theory has mostly been used as a source of inspiration in robotics. Most of the studies reviewed preferred to refer to J.J. Gibson's original ideas as formulated in his books, ignoring modern discussions on the concept. As a result, only certain aspects of the theory have been used, and no attempts to consider the implications of the whole theory towards autonomous robot control have been made.

The affordance-based robot control architecture has provided a proof-of-concept implementation of how the formalism can be implemented and used in real-world. The experimental results obtained from partial implementations of the architecture has also provided convincing examples that the approach generates results that are beyond the current state-of-the-art in the field.

top of page

5b. Representation for deliberation and planning:

AI has envisaged for a long time robots acting as aided by symbolic plans, which are generated on the fly by state-of-the-art action planners. However, plan-based robot control has turned out to be difficult; one bag of problems associated with it is anchoring of objects in sensor data and of plan operators in physical action at execution. Using affordances as a first-class citizen in the control architecture has shed some new light upon the problem. Whenever the planner's symbolic domain representation can be made to include actions that correspond to affordances (like, e.g., a 'lift' action in the MACS scenario, derived from the 'liftability' affordance), then grounding its execution comes practically for free. The same is true for representations of affordances that appear in the preconditions of symbolic actions: Determining their validity at execution means to attend to the respective affordance.

As a result from our study, it turns out that affordance-based and plan-based control appear to be easy fellows in a joint robot control architecture. This is not too surprising, given that they both deal with action, under different perspectives. We are not aware, however, that this has been explored in robotics.

top of page

5c. Perception:

Affordance perception has been considered so far only in the frame of functional object recognition and in the context of action based perception. A key aspect that has been neglected in previous approaches is the novel type of generalisation that refers to the interaction of an agent with parts of its environment. Visual features should not be selected by the engineer but should be extracted for their relevance to predict opportunities for action. For this purpose we considered in MACS numerous types of 2D and 3D features and demonstrated that machine learning methodology can determine a meaningful selection of features that generalise across various appearances while they at the same time characterise a specific physical property which can be exploited in the predicted interaction. A particular contribution in MACS is not only to argue but also to verify that affordances are not associated to objects - a notion which is derived from an ontology that is grounded in human perception - but can be related to arbitrary chunks of information, such as visual regions, that possess a predictive nature in estimating future outcomes of an agent's interaction with the world and its physical properties. The predictive nature of affordance cues requires to perform feature selection with respect to the goal-directedness of behaviours, the relevance of the outcome: irrelevant outcomes of actions are negligible for the agent and therefore negligible for perception. Consequently we introduced a framework of feature selection and extraction in the context of the utility of actions through the association of behaviours to reward functions, rendering classification of features impacted by utility instead of identity as it is done in classical recognition frameworks. The investigations performed in MACS contributed to pave the way for a new understanding of machine perception, moving away from manual labelling of image databases towards the autonomous emergence of symbols that are grounded in sensory-motor interactions with the real world and the associated relevance for the agent.

top of page

5d. Learning:

The learning approach that was developed within MACS and used in the Learning Module of the architecture is an approach to acquire knowledge about relations that determine the interaction possibilities between an agent and its environment. Affordances are thus represented in form of entity-action-outcome triples and thus refer to both, the agent (its capabilities) and its environment, as initially claimed by Gibson. In our work we describe how an artificial agent, starting with basic interactions and using more and more complex interactions, acquires this knowledge by using its gathered experience. These experiences are generalized by the agent, enabling it to act also in novel situations.

The approach is a general framework in which unsupervised learning methods as well as reinforcement learning methods are utilized. Hence the approach does not depend on a special learning mechanism and is thus open to future developments. Statistical approaches as well as traditional and novel self-organizing algorithms (as neural networks, Multi-SOMs [92], and ROLF [93]) are used. The developed learning approach clearly addresses and defines how an artificial agent can gain the necessary knowledge required for the purposeful usage of affordances and the combination of different learning methods within this approach instead of using a monolithic learning method, which is novel compared to the state of the art. The approach starts with an initial set of reflex like actions and is designed to be able to deal with a growing set of (learned) actions. Thereby the approach is not limited to a special kind of actions. Therefore the MACS affordance learning approach closes the developmental gap between an 'infant agent' and an agent that is able to use affordances purposefully. This provides a strong basis for using affordances in robotics to gain a more flexible and robust goal driven behaviour in artificial agents; and since the approach is a general concept that can be used to enable a wide range of artificial agents to learn affordances within its environment, it is not limited to a special scenario or agent. The final software implemented within MACS, demonstrated through a number of experiments, shows the viability of the approach in practice.

top of page

5e. Architecture:

We would like to note that before MACS, the affordance theory of J.J. Gibson was mostly used either as a source of inspiration for isolated features of control in autonomous robotics, or the concept of affordances was used to merely describe the behaviour of a robotic system. Consequently, only certain aspects of the theory were used, and no attempts to consider the implications of the whole theory towards autonomous robot control were made. In this sense, the development of a complete affordance-inspired robot control architecture that is designed to learn, detect, and use the affordances in the environment is an important contribution of MACS to the field.

top of page

6. References

References of type [Dx.y.z] refer to public deliverables of MACS. p

[Bær02]	Bærentsen, K.B. & Trettvik, J. An activity theory approach to affordance. In NordiCHI '02: Proc. 2nd Nordic conf. on Human-computer interaction, pp. 51-60. ACM Press, 2002.
[Cal03]	Calderon, C.A.A. & Hu, H. Goal and actions: Learning by imitation. In Proceedings of the Second International Symposium on Imitation in Animals and Artifacts (AISB 03), pp. 179-182, Aberystwyth, Wales, UK, 2003.
[Duc98]	Duchon, A.P., Warren, W.H., & Kaelbling, L.P. Ecological robotics. In Adaptive Behavior, Special Issue Biologically Inspired Models of Spatial Navigation, 6:3/4, pp. 473-507, 1998.
[Edw03]	Edwards, M.G., Humphreys, G.W., & Castiello, U. Motor facilitation following action observation: a behavioral study in prehensile action. In Brain Cognition, 53, pp. 495-502, 2003.
[Gib77]	Gibson, J.J. The theory of affordances. In R. Shaw and J. Bransford, editors, Perceiving, Acting and Knowing, pp. 67-82. Wiley, New York, 1977.
[Gib79]	Gibson, J.J. The Ecological Approach to Visual Perception. Boston, Houghton Mifflin, 1979.
[Mur99]	Murphy, R.R. Case studies of applying Gibson's ecological approach to mobile robots. IEEE Transactions on Systems, Man, and Cybernetics, 29(1):105-111, 1999.
[MDor00]	MacDorman, K.F. Responding to affordances: Learning and projecting a sensorimotor mapping. In Proc. of 2000 IEEE International Conference on Robotics and Automation, pp. 3253-3259, San Fransisco, California, USA, 2000.
[Fit03]	Fitzpatrick, P.; Metta, G.; Natale, L.; Rao, S.; Sandini, G. Learning about objects through action � initial steps towards artificial cognition. In Proceedings of the 2003 IEEE International Conference on Robotics and Automation, ICRA, pp. 3140-3145, 2003.
[Sto05a]	Stoytchev, A. Toward learning the binding affordances of objects: A behaviour-grounded approach. In Proceedings of AAAI Symposium on Developmental Robotics, pp. 17-22. Stanford University, March 21-23, 2005.
[Sto05b]	Stoytchev, A. Behaviour-grounded representation of tool affordances. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 3071�3076, 2005. Conference: Barcelona, Spain, April 18-22, 2005.
[Sah07]	Sahin, E.; Cakmak, M.; Dogar, M.R.; Ugur, E. and Ucoluk, G. To afford or not to afford: A new formalization of affordances towards affordance-based robot control. Adaptive Behavior, volume 15, number 4, pp. 447-472, 2007.
[Rom08a]	Rome, E., Hertzberg, J., & Dorffner, G. (eds.). Towards Affordance-Based Robot Control, Proceedings of the Dagstuhl seminar 06231, Springer LNAI 4760, Berlin, February 2008. Conference: Schloss Dagstuhl International Conference and Research Center for Computer Science in Wadern (Germany), http://www.dagstuhl.de/06231/, June 5-9, 2006.
[Rom08b]	Rome, E., Paletta, L., Sahin, E., Dorffner, G., Hertzberg, J. et al. The MACS project: An approach to affordance-based robot control, in: Towards Affordance-Based Robot Control, Proceedings of the Dagstuhl seminar 06231, pp. 173-210, Springer LNAI 4760, Berlin, February 2008. Conference: Schloss Dagstuhl International Conference and Research Center for Computer Science in Wadern (Germany), http://www.dagstuhl.de/06231/, June 5-9, 2006.

top of page