Fast reinforcement learning for simple physical robots

In the past few years, the field of autonomous robot has been rigorously studied and non-industrial applications of robotics are rapidly emerging. One of the most interesting aspects of this field is the development of the learning ability which enables robots to autonomously adapt to given environments without human guidance. As opposed to the conventional methods of robots’ control, where human logically design the behavior of a robot, the ability to acquire action strategies through some learning processes will not only significantly reduce the production costs of robots but also improves the applicability of robots in wider tasks and environments. However, learning algorithms usually require large calculation cost, which make them unsuitable for robots with limited resources. In this study, we propose a simple two-layered neural network that implements a novel and fast Reinforcement Learning. The proposed learning method requires significantly less calculation resources, hence is applicable to small physical robots running in the real world environments. For this study, we built several simple robots and implemented the proposed learning mechanism to them. In the experiments, to evaluate the efficacy of the proposed learning mechanism, several robots were simultaneously trained to acquire obstacle avoidance strategies in the same environment, thus, forming a dynamic environment where the learning task is substantially harder than in the case of learning in a static environment and promising result was obtained.     

Mathematical modeling and computer simulation of a robotic rat pup

In this paper, we describe the mathematics and computer implementation of a robotic rat pup simulation. Our goal is to understand neurobehavioral principles in a mammalian model organism—the Norway rat pup (Rattus norvegicus). Our approach is unique in that animal, simulation, and robot studies occur in parallel and inform each other. Behavior is dependent on the nervous system, body morphology, physiology, environment, and the interactions among these elements. Autonomous robotics hardware models and their associated simulations allow the possibility of systematically manipulating variables in each of these elements in ways that would be impossible using live animals. Specifically, we describe the development and validation of a Newtonian-dynamics-based simulation of a robotic rat pup, including mathematical formulation and computer implementation. The computer simulation consists of three distinct components that interact to simulate robotic behavior: (1) dynamics of the robotic rat pup itself, including sensors and actuators, (2) environmental coupling dynamics of the robot arena with the robotic rat pup, and (3) the robot control algorithms as implemented on the physical robot. The mathematical formulation, software implementation, model identification, model validation, and an application example are all described.     

A cellular automata ant memory model of foraging in a swarm of robots

This study proposes a cellular automata ant memory model (CAAM) that controls a robot swarm when undertaking the foraging task in a previously known environment with nests. The floor field is well known to all robots, which share the same environment, communicating through the inverted pheromone. This substance is deposited by each swarm robot over every step in searching, which results in a repulsive force between team members. Besides, a short-term memory inspired by Tabu Search is applied to enable robots to remember their last positions and to avoid useless explorations. On the other hand, homing is based on the behavior observed in pedestrian evacuation, resulting in an attractive force through the nests. Moreover, a dynamic information is used to avoid queues of robots and bottlenecks next to the nests. Each robot step is a first choice movement with a stochastic conflict solver, which results in a non-deterministic characteristic to the model. The proposed model was implemented and submitted to several simulations to evaluate its resultant behavior. Different environmental conditions were employed to refine its intrinsic parameters. The results shown that the proposed model is able to perform the foraging task in a competitive way: in searching the swarm perform a good environment coverage and in homing robots are able to find the most adequate nests.     

Robust position/force control of multiple robots using neural networks

The control of multiple redundant robots, whose end-effectors grasp an object, involves complex control tasks. First, the multiple robotic system, for a cooperative task, forms closed kinematic chains that impose additional kinematic and dynamic constraints. Second, the interactive actions among the robots through the object lead to the essential need to control position and interactive force, simultaneously. Finally, the structured and unstructured uncertainties of the system may cause the system to be unstable. In this paper, a robust controller, which compensates the uncertainties of the dynamic system of the multiple robotic system, is presented in order to obtain good tracking performance of position and force, simultaneously, while satisfying the constraint conditions among the robots. A neural network architecture is proposed as one approach to the design and implementation of the robust controller. In particular, an on-line learning rule is provided for reportedly assigned tasks so that the system is robust to the structured/unstructured uncertainties; and the controller adjusts itself repeatedly to improve the performance progressively for each repeated task.     

The synthetic teammate project

The main objective of the Synthetic Teammate project is to develop language and task enabled synthetic agents capable of being integrated into team training simulations. To achieve this goal, the agents must be able to closely match human behavior. The initial application for the synthetic teammate research is creation of an agent able to perform the functions of a pilot for an Unmanned Aerial Vehicle (UAV) simulation as part of a three-person team. The agent, or synthetic teammate, is being developed in the ACT-R cognitive architecture. The major components include: language comprehension and generation, dialog management, agent-environment interaction, and situation assessment. Initial empirical results suggest that the agent-environment interaction is a good approximation to human behavior in the UAV environment, and we are planning further empirical tests of the synthetic teammate operating with human teammates. This paper covers the project’s modeling approach, challenges faced, progress made toward an integrated synthetic teammate, and lessons learned during development.     

Robot learning schemes that trade motion accuracy for command simplification

This study was inspired by the human motor control system in its ability to accommodate a wide variety of motions. By contrast, the biologically inspired robot learning controller usually encounters huge learning space problems in many practical applications. A hypothesis for the superiority of the human motor control system is that it may have simplified the motion command at the expense of motion accuracy. This tradeoff provides an insight into how fast and simple control can be achieved when a robot task does not demand high accuracy. Two motion command simplification schemes are proposed in this paper based on the equilibrium-point hypothesis for human motion control. Investigation into the tradeoff between motion accuracy and command simplification reported in this paper was conducted using robot manipulators to generate signatures. Signature generation involves fast handwriting, and handwriting is a human skill acquired via practice. Because humans learn how to sign their names after they learn how to write, in the second learning process, they somehow learn to trade motion accuracy for motion speed and command simplicity, since signatures are simplified forms of original handwriting. Experiments are reported that demonstrate the effectiveness of the proposed schemes.     

Solving a robotic assembly line balancing problem using efficient hybrid methods

In this paper we are studying a robotic assembly line balancing problem. The goal is to maximize the efficiency of the line and to balance the different tasks between the robots by defining the suitable tasks and components to assign to each robot. We are interested in a robotic line which consists of seizing the products on a moving conveyor and placing them on different location points. The performances evaluations of the system are done using a discret event simulation model. This latter has been developed with C++ language. As in our industrial application we are bounded by the execution time, we propose some resolution methods which define the suitable component and point positions in order to define the strategy of pick and place for each robot. These methods are based on the ant colony optimization, particle swarm optimization and genetic algorithms. To enhance the quality of the developed algorithms and to avoid local optima, we have coupled these algorithms with guided local search. After that, an exact method based on full enumeration is also developed to assess the quality of the developed methods. Then, we try to select the best algorithm which is able to get the best solutions with a small execution time. This is the main advantage of our methods compared to exact methods. This fact represents a great interest taking in consideration that the selected methods are used to manage the functioning of real industrial robotic assembly lines. Numerical results show that the selected algorithm performs optimally for the tested instances in a reasonable computation time and satisfies the industrial constraint.     

Robot Manipulations: A Synergy of Visualization, Computation and Action for Spatial Instruction

This article considers the use of a learning environment, RoboCell, where manipulations of objects are performed by robot operations specified through the learner's application of mathematical and spatial reasoning. A curriculum is proposed relating to robot kinematics and point-to-point motion, rotation of objects, and robotic assembly of spatial puzzles. Various instructional methods are supported by the RoboCell robot system, such as interactive demonstrations, modeling, computer simulations and robot operations, providing diverse activities in spatial perception, mental rotation and visualization. Pre-course and post-course tests in two middle schools and a high school indicated significant student progress in the tasks related to the categories of spatial ability which were practiced in the course. This revised version was published online in July 2006 with corrections to the Cover Date.     

Modelling a hormone-inspired controller for individual- and multi-modular robotic systems

For all living organisms, the ability to regulate internal homeostasis is a crucial feature. This ability to control variables around a set point is found frequently in the physiological networks of single cells and of higher organisms. Also, nutrient allocation and task selection in social insect colonies can be interpreted as homeostatic processes of a super-organism. And finally, behaviour can also represent such a control scheme. We show how a simple model of hormone regulation, inspired by simple biological organisms, can be used as a novel method to control the behaviour of autonomous robots. We demonstrate the formulation of such an artificial homeostatic hormone system (AHHS) by a set of linked difference equations and explain how the homeostatic control of behaviour is achieved by homeostatic control of the internal ‘hormonal’ state of the robot. The first task that we used to check the quality of our AHHS controllers was a very simple one, which is often a core functionality in controller programmes that are used in autonomous robots: obstacle avoidance. We demonstrate two implementations of such an AHHS controller that performs this task in differing levels of quality. Both controllers use the concept of homeostatic control of internal variables (hormones) and they extend this concept to also include the outside world of the robots into the controlling feedback loops: As they try to regulate internal hormone levels, they are forced to keep a homeostatic control of sensor values in a way that the desired goal ‘obstacle avoidance’ is achieved. Thus, the created behaviour is also a manifestation of the acts of homeostatic control. The controllers were evaluated using a stock-and-flow model that allowed sensitivity analysis and stability tests. Afterwards, we have also tested both controllers in a multi-agent simulation tool, which allowed us to predict the robots' behaviours in various habitats and group sizes. Finally, we demonstrate how this novel AHHS controller is suitable to control a multi-cellular robotic organism in an evolutionary robotics approach, which is used for self-programming in a gait-learning task. These examples shown in this article represent the first step in our research towards autonomous aggregation and coordination of robots to higher-level modular robotic organisms that consist of several joined autonomous robotic units. Finally, we plan to achieve such aggregation patterns and to control complex-shaped robotic organisms using AHHS controllers, as they are described here.     

Dynamics and stability of a hybrid serial-parallel mobile robot

Kinematics, dynamics, and stability analysis of a hybrid serial-parallel wheeled mobile robot is detailed in this paper. Privileging the advantages of both serial and parallel robots, the suggested structure will provide higher stability for heavy object manipulation by a mobile robotic system. The proposed system is made of a differentially-driven wheeled platform, a planar parallel manipulator, which is called here as star-triangle (ST) mechanism, and a serial Puma-type manipulator arm. In order to develop a comprehensive kinematics model of the robot; first it is divided into three modules, i.e. a mobile platform, a parallel ST mechanism, and a serial robot. Next, a closed-form dynamics model is derived for the whole hybrid system based on a combined Newton–Euler and Lagrange formulation. Then, a careful validation procedure is presented to verify the obtained dynamics model. Finally, using the new postural stability metric named as moment-height stability (MHS), the important role of the parallel ST mechanism for stabilizing the mobile robotic system is demonstrated. The obtained results show that the proposed hybrid serial–parallel arrangement effectively enhances the tip-over stability of the overall mobile robotic system. Hence, it can be successfully exploited to prevent tip-over instability particularly during heavy object manipulation tasks.     

A genetic algorithm for robotic assembly line balancing

Flexibility and automation in assembly lines can be achieved by the use of robots. The robotic assembly line balancing (RALB) problem is defined for robotic assembly line, where different robots may be assigned to the assembly tasks, and each robot needs different assembly times to perform a given task, because of its capabilities and specialization. The solution to the RALB problem includes an attempt for optimal assignment of robots to line stations and a balanced distribution of work between different stations. It aims at maximizing the production rate of the line. A genetic algorithm (GA) is used to find a solution to this problem. Two different procedures for adapting the GA to the RALB problem, by assigning robots with different capabilities to workstations are introduced: a recursive assignment procedure and a consecutive assignment procedure. The results of the GA are improved by a local optimization (hill climbing) work-piece exchange procedure. Tests conducted on a set of randomly generated problems, show that the Consecutive Assignment procedure achieves, in general, better solution quality (measured by average cycle time). Further tests are conducted to determine the best combination of parameters for the GA procedure. Comparison of the GA algorithm results with a truncated Branch and Bound algorithm for the RALB problem, demonstrates that the GA gives consistently better results.     

Cognitive modeling of socially transmitted affordances: a computational model of behavioral adoption tested against archival data from the Stanford Prison Experiment

Social learning and adoption of new affordances govern the rise of new a variety of behaviors, from actions as mundane as dance steps to those as dangerous as new ways to make improvised explosive device (IED) detonators. Traditional diffusion models and social network structures fail to adequately explain who would be likely to imitate new behavior and why some agents adopt the behavior while others do not. To address this gap, a cognitive model was designed that represents well-known socio-cognitive factors of attention, social influence, and motivation that influence learning and adoption of new behavior. This model was implemented in the Performance Moderator Function Server (PMFServ) agent-based cognitive architecture, enabling the creation of simulations where affordances spread memetically through cognitive mechanisms. This approach models facets of behavioral adoption that have not been explored by existing architectures: unintentional learning, multi-layered social and environmental attention cues, and contextual adoption. To examine the effectiveness of this model, its performance was tested against data from the Stanford Prison Experiment collected from the Archives of the History of American Psychology.     

Where Forward-Looking and Backward-Looking Models Meet

The present paper begins by deriving an instantaneous formulation for the backward-looking (reinforcement based learning) satisfaction balance model of Gray and Tallman (1984). This model is then used to generate interactional data from four simulated agents in a network interaction experiment. Because this initial model does not generate stable interaction structures in the network experiment, it is altered step by step in the direction of a forward-looking (agent with goals) model that has been shown to generate such stable interaction structures. The purpose of the modifications are to learn what aspects of the forward-looking model are needed to evolve a stable interaction structure, and to learn how these aspects may be incorporated into a model that remains essentially reinforcement based.     

Neuro-fuzzy technique for navigation of multiple mobile robots

In this paper, navigation techniques for several mobile robots are investigated in a totally unknown environment. In the beginning, Fuzzy logic controllers (FLC) using different membership functions are developed and used to navigate mobile robots. First a fuzzy controller has been used with four types of input members, two types of output members and three parameters each. Next two types of fuzzy controllers have been developed having same input members and output members with five parameters each. Each robot has an array of sensors for measuring the distances of obstacles around it and an image sensor for detecting the bearing of the target. It is found that the FLC having Gaussian membership function is best suitable for navigation of multiple mobile robots. Then a hybrid neuro-fuzzy technique has been designed for the same problem. The neuro-fuzzy technique being used here comprises a neural network, which is acting as a pre processor for a fuzzy controller. The neural network considered for neuro-fuzzy technique is a multi-layer perceptron, with two hidden layers. These techniques have been demonstrated in simulation mode, which depicts that the robots are able to avoid obstacles and reach the targets efficiently. Amongst the techniques developed neuro-fuzzy technique is found to be most efficient for mobile robots navigation. Experimental verifications have been done with the simulation results to prove the authenticity of the developed neuro-fuzzy technique.     

Perspectives on Constructive and Functional Optimizing of a Serial Robot with Four Degrees of Freedom Destined for Special Applications

This paper deals with aspects on the dynamic modulation of the robots mechanical structure, using Lagrangian formalism, choosing the adequate DC servomotors, translation modules constituting the TTRT robot, the constructive solution and modelling the three translation subassemblies of the studied robot, with the intention that, in the end, based on a dynamic-organologic algorithm, a functional optimization of the robot be brought out within a workcell destined for special applications, so that the energetic consumption be as low as possible. This paper also presents the organological calculi and solutions for the efficient design of modules specific to mechanical structures of industrial serial-modular robots. (© 2012 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

The Relationships between the Shape of the Workspace and GeometricalDimensions of Parallel Manipulators

In this paper, a new simulation program for the workspace analysis of parallel robots is presented. The system consists of modules, which enable a complex study of the parallel structures. A special method for generating the workspace of the parallel structures is used. The simulation program offers the possibility to generate automatically the virtual model of the parallel robot using the facilities of a 3D modeling software. The structure geometric parameters, which could be modified within the modeling software, influence the simulation environment. These facilities enable to establish the relationships between the workspace shape, elements links and geometrical parameters of the structure. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

A social approach for target localization: simulation and implementation in the marXbot robot

Foraging is a common benchmark problem in collective robotics in which a robot (the forager) explores a given environment while collecting items for further deposition at specific locations. A typical real-world application of foraging is garbage collection where robots collect garbage for further disposal in pre-defined locations. This work proposes a method to cooperatively perform the task of finding such locations: instead of using local or global localization strategies relying on pre-installed infrastructure, the proposed approach takes advantage of the knowledge gathered by a population about the localization of the targets. In our approach, robots communicate in an intrinsic way the estimation about how near they are from a target; these estimations are used by neighbour robots for estimating their proximity, and for guiding the navigation of the whole population when looking for these specific areas. We performed several tests in a simulator, and we validated our approach on a population of real robots. For the validation tests we used a mobile robot called marXbot. In both cases (i.e., simulation and implementation on real robots), we found that the proposed approach efficiently guides the robots towards the pre-specified targets while allowing the modulation of their speed.     

Path planning in formation and collision avoidance for multi-agent systems

This paper investigates, in a centralized manner, the motion planning problem for a team of unicycle-like mobile robots in a known environment. In particular, a multi-agent collision-free patrolling and formation control algorithm is presented, which combines outcomes of: (i) stability analysis of hybrid systems, (ii) algebraic geometry, and (iii) classical potential functions. The objective is achieved by designing a Lyapunov-based hybrid strategy that autonomously selects the navigation parameters. Tools borrowed from algebraic geometry are adopted to construct Lyapunov functions that guarantee the convergence to the desired formation and path, while classical potential functions are exploited to avoid collisions among agents and the fixed obstacles within the environment. The proposed navigation algorithm is tested in simulation and then validated by using the robots of a remote accessible robotic testbed.     

Robust Optimal Trajectory Planning for Robots by Stochastic Optimization

In the optimal control of industrial, field or service robots, the standard procedure is to determine first offline a reference trajectory and a feedforward control, based on some selected nominal values of the unknown stochastic model parameters, and to correct then the inevitable and increasing deviation of the state or performance of the robot from the prescribed state or performance of the system by online measurement and control actions. Due to the stochastic variations of the model parameters, increasing measurement and correction actions are needed during the process. By optimal stochastic trajectory planning (OSTP), based on stochastic optimization methods, the available a priori and sample information about the robot and its working environment is incorporated into the control process. Consequently, more robust reference trajectories and feedforward controls are obtained which cause much less online control actions. In order to maintain a high quality of the guiding functions, the reference trajectory and the feedforward control can be updated at some later time points such that additional information about the control process is available. After the presentation of the Adaptive Optimal Stochastic Trajectory Planning (AOSTP) procedure based on stochastic optimization methods, several numerical techniques for the computation of robust reference trajectories and feedforward controls under real-time conditions are presented. Additionally, numerical examples for a Manutec r3 industrial robot are discussed. The first one demonstrates real-time solutions of (OSTP) based on a sensitivity analysis of a before-hand calculated reference trajectory. The second shows the differences between reference trajectories based on deterministic methods and the stochastic methods introduced in this paper. Based on simulations of the robots behavior, the increased robustness of stochastic reference trajectories is demonstrated.