2.1. Theoretical Basis for Conceptual Design
Aed model
considered below unites the codes of the two level system [
5] and general systems theory [
11], numeric positional codes
, geometry, and cybernetic methods. Aed is a standard element of Hierarchical Systems [
5,
6,
7,
8,
9], which implements the general laws of systems organization on each level and the inter-level connections (see
Figure 1). Aed
contains
and
models connected by coordinator
:
where
is the dynamic representation of the level
system in its environment
;
is the system structure;
is the coordinator; ℓ ∈
Ls,
Ls is numerical positional system. Aed scheme is given in
Figure 1. Connected dynamic representations
of aed elements, i.e., object
, processes
and environment
are presented in
form:
where
and
are reactions and state transition functions respectively,
is state,
is input,
is output, and
is time of level
. Object
, processes
{
} and environment
are connected by their states, inputs and outputs [
5,
6,
7,
8,
9].
The structure of system
is defined in the following way:
and
is as follows:
where
is a coordinator,
are aggregated dynamic models of subsystems
, and
are structural connections.
is the set of
subsystems and their structural connections
coordinated with external ones
.
Being the main element of hierarchical systems, the coordinator performs the design and control tasks on its strata [
5,
6,
7,
8,
9]. The coordinator is presented according to aed model (1) in the form (5):
where
is the dynamic realization of
,
is the structure of coordinator
, and
is the coordinator control unit. Coordinator
is constructed recursively, and performs the design and control tasks on its selection, learning and self-organization strata [
5,
6,
7,
8,
9].
Geometrical signs and metric characteristics
μℓ of systems
Sℓ being designed and controlled are defined in the framework of aed formal model in codes of numeric positional system
[
5,
6,
7,
8,
9]. The main of them are constructive dimension
δℓ and connection defect
ξℓ. For instance, the constructive dimension
δℓ is presented in code
as follows:
where
and
are constructive dimensions of
and
respectively.
This representation of geometrical information about systems Sℓ being designed allows execution of all computer operations with geometric images as operations with numeric codes.
Aed formal model briefly described above presents a theoretical basis for the BCI biomechatronic system conceptual design and control.
2.2. Conceptual Model for BCI System Design
The conceptual model of the BCI system is briefly presented below, as an example of a biomechatronic system description in the theoretical basis of hierarchical systems (HS). The object under consideration in this work is the BCI system (
Figure 2), with its measuring, computer, man-operator, and mobile robot subsystems, developed in AGH University of Science and Technology, Poland [
1,
2]. Systemic conceptual model of the BCI system is given in aed form (6) of HS:
where,
is an aggregated dynamic representation of BCI,
is the system structure,
is BCI system coordinator (design & control system), and
is the index of level.
Structural scheme of the BCI system being designed is presented in
Figure 2 in agreement with aed diagram (
Figure 1). According to (3,4), BCI
system structure contains coordinator
, the set of sub-systems
and their structural connections
. The BCI subsystems
presented in
form (2) are:
: man-operator; : measuring system;
: computer control system; : mobile robot.
BCI system coordinator is presented by man-coordinator and distributed design and control units. Computer control BCI subsystem partially performs coordinator functions as well.
In their turn, each subsystem iSℓ−1 of level ℓ−1 has its own structural elements—lower level ℓ−2 subsystems presented in aggregated dynamic ijωℓ−2 form. For the Mindstorms NXT 2.0 mobile robot 4ωℓ−1 they are executive subsystem, i.e., wheels 41ωℓ−2 and servos 42ωℓ−2, and robot’s own NXT 2.0 control unit 43ωℓ−2. Subsystems of the mobile robot are connected by their structural connections σγℓ−2. For example, servos 42ωℓ−2 and wheels 41ωℓ−2 are connected by gear transmission, which is the structural connection of them, and at the same time is subsystem of level l−3, i.e., structural element of servo 42ωℓ−2. Similarly, the higher level subsystems are connected by their common parts, i.e., structural connections σγℓ−1 that are the elements of lower levels. Man-operator 1ωℓ−1 and measuring subsystem 2ωl−1 are connected by their common elements—electrodes attached to human’s head.
Aggregated dynamic realizations ω
ℓ−1, i.e., dynamic models
i(ρ,φ)ℓ−1 of BCI subsystems are formed after the definition of their inputs–outputs according to each concrete sub-process they execute. As for the mobile robot
4Sℓ−1 subsystem, concerning its motion process, the realization
4ωℓ−1 used at the conceptual design phase in
(ρ,φ) form (2) can be presented in form (8) at the detailed design phase as the following kinematics equations [
12]:
which connects linear
and angular
velocities of robot center point
P(
x,
y) (located in the middle of the axis connecting driven robot wheels) as the output
4Yℓ−1 with the input
4Xℓ−1, i.e., linear velocities of the centers of the left
vl and right
vr wheels of robot.
a is the distance between the centers of the left and right wheels, and φ is the angle between
Xg and
Xr axes of the global
g and the local robot
r coordinate systems respectively (
Figure 3). The robot designed has the independent drives of both wheels and Instantaneous Center of Curvature (ICC), around which the rotary motion is performed by robot. ICC is placed in the line connecting the centers of the two wheels. For the case of the robot motion shown in
Figure 3, ICC is separated from point
P(
x,
y) by the distance
R. The rotation around the ICC point takes place if speed values of each wheel are different.
The model of robot kinematics was implemented within the frameworks of Matlab/Simulink program environment for performing a kinematic task at the detailed design phase. The general view of the model is presented in
Figure 4. The inputs of RobotKinematicModel block are linear velocities of the centers of the left
vl and right
vr wheels of the robot respectively. The outputs are point
P coordinates (
x,
y), ICC coordinates
xICC and
yICC, distance
R, and angle φ value phi (
Figure 4a). The detailed structure of the
RobotKinematicModel block is shown in
Figure 4b, which illustrates the graphical representation of the robot kinematics equations describing the robot’s location.
At the detailed design phase, simulation studies of the mobile robot movement were carried out. For this purpose, a special environment was created to animate the movement of the robot. The robot’s movement was set by the linear velocity value of the left
vl and the right
vr of the wheel.
Figure 5a presents the mentioned linear velocities of both wheels. Vertical bold lines indicate two time moments (
t1 = 2.49 s and
t2 = 9.19 s).
Figure 5b,c show pictures from the simulation showing the current position of the robot along with the trajectory of the point
P (black) and the point
Pp (blue) for
t1 and
t2 time moments respectively. The green lines represent linear velocity vectors with the beginnings in the center of each wheel. The lengths of the vectors represent the values of the current linear velocity shown in the upper right corners of the simulation
Figure 5b,c. Additionally, the current values of the following parameters are shown: simulation time (
t), current position of the
P point, i.e.,
x,
y coordinates, and angle
φ value. The current position of the ICC and radius
R (red line) are also presented.
As for the robot dynamics, aggregated dynamic model
of the MR can be presented at the phase of the detailed design like in [
13] or in the following form:
State equation in (9) corresponds to the state transition function in MR conceptual model (2) and the second output equation corresponds to the reaction . Vectors x, y, u and matrices A, B, C must be predefined. In the case of the mobile robot motion, the elements of states vector x = [x1 x2]T are the displacement x1 and velocity x2. Mobile robot kinematics and dynamics tasks, briefly described above, belong to analysis design task of the BCI subsystem being designed, i.e., mobile robot in our case.
Concerning the man-operator subsystem, its model can be constructed on its inputs , i.e., visual stimuli signals, perceived by human eyes; its outputs , i.e., EEG signals recorded by measuring system , taking into consideration human’s physiological state .
Measuring subsystem
purpose is EEG signals registration, processing and data transmitting to BCI control system by TrueScan 24 subsystem. TruScan Acquisition program is used as software means to perform impedance measurements. The electroencephalographic (EEG) signal was recorded using the EEG cap of Deymed Diagnostic company (
Figure 6a), containing 19 chlorine silver electrodes (Ag/AgCl) arranged according to the International System 10–20. In addition, the cap contained a reference electrode and a grounding electrode (
Figure 6b). The cap is placed on the head of the subject, i.e., man-operator, being examined (
Figure 6c). EEG gel application to the measuring electrodes allowed to maintain electrodes impedance at 2–5 kΩ level during the conducted tests.
The BCI environment has its own structure and contains the following sub-systems: panel which generates light stimuli perceived by man-operator’s eyes; floor area where the mobile robot moves; man-coordinator , which places electrodes on man-operator’s head and attaches the measuring system; the higher level l+1 design and control system .
The BCI system contains
sub-processes, which are the parts of the higher-level process
(see
Figure 1) performed by the entire BCI system: (
) ionic current flows generation within the neurons of the human brain, performed by the man-operator, (
) EEG signal acquisition by the measuring system; (
) data analysis and control signals generation for the mobile robot performed by computer subsystem; (
) mobile robot motion. Each process contains its lower-level sub-processes, e.g., (
) process of computer subsystem contains low-level ℓ−1 process of data reading
, digital filtering
, Fast Fourier Transformation
, signal spectral analysis
, frequency verification
, and generation of control signals for the mobile robot
. All these processes are described according to (2).
So, all the subsystems of the BCI biomechatronic system, i.e., man-operator , measuring , computer , and mobile robot systems have their aggregated dynamic and structural descriptions. All the connected descriptions of the subsystems and their processes are presented in the knowledge base of the coordinator that performs the design and control tasks, connecting in this way structure and dynamic realization of the BCI system being coordinated.
BCI coordinator
is described according to (5) and implemented by the human–computer design and control BCI subsystems including the MR controller. The coordinator maintains BCI functional modes by the man-operator and control system and performs the design process by the higher level CAD system. The BCI control system is described in the following
Section 2.3.
2.3. Design of BCI Control System
The design process of BCI control system of the mobile robot (MR) also contains two stages, i.e., conceptual and detailed design phases. The conceptual model of the BCI control system is presented in form (5) of HS coordinator (
) briefly described above in
Section 2.1 as well as in another work [
2]. Aggregated dynamic model
of the BCI control system (
) is built on the relation of the control system inputs
, outputs
and states
, presented in the following
form:
The inputs
X0 and outputs
Y0 of the coordinator of
ℓ − 1 level are defined on the sets of its coordination signals
G and feedbacks
W as follows:
As for the control system of Mindstorms mobile robot (
Figure 7), it is presented by the internal control unit NXT 2.0 (which consists of ARM7 and AVR microcontrollers) and BCI control PC in the form of the coordinator according to the scheme presented in
Figure 1 (
Section 2.1).
In
Figure 7,
Gℓ−1 are coordination signals from the internal NXT 2.0 control unit of the MR to MR
oS4 actuating systems, i.e., servos and wheels. The MR position measured by MR encoder with 1 degree precision is a feedback
from MR actuating systems
oS4 to NXT 2.0 control unit
4S0;
Wℓ is a feedback from MR, i.e., NXT 2.0 control unit, to the BCI PC control system
S0 of higher level
ℓ.
Gℓ are the control signals from control PC
S0 to mobile robot
4Sℓ−1. The NXT 2.0 controller
4S0 is directly connected by its input
Xℓ = (
Gℓ,
Wℓ−1) and output
Yℓ = (
Gℓ−1,
Wℓ) with the MR actuating systems and with the higher level
ℓ coordinator
S0, i.e., BCI PC design and control system, where (
Gℓ−1,
Wℓ−1) and (
Gℓ,
Wℓ) are lower level
ℓ−1 and higher level
ℓ coordination
G and feedback
W signals respectively.
Communication of the BCI PC control system S0 with lower ℓ−1 level NXT 2.0 control unit 4S0 of Mindstorms mobile robot is realized on the program level with the help of RWTH—Mindstorms NXT Toolbox integrated in MATLAB environment of the control PC. Control Gℓ and information Wℓ signals are transmitted with 12 Mbit/s speed. The transmission is possible owing to the implementation of Bluethoot 2.0 technology in NXT 2.0 control unit.
Concrete values of the BCI PC control system
inputs
, outputs
and states
are defined and detailed in the design phase and described below. A concrete model of BCI control system
was implemented in the form of finite state machine [
14] at the detailed design phase according to its conceptual model. Finite state machine model of BCI control system (CS) was implemented within the frameworks of Matlab program system using Simulink Stateflow module. To implement control algorithms by generating control codes, Matlab, Simulink, and Embedded Coders were used. The proposed control system developed in Matlab is presented in
Figure 8. The main element of the BCI control system is
MainController unit, which realizes logic laws of control of TruStan 24 electroencephalograph (EEG) unit, the light stimulator and the mobile robot. According to the BCI CS conceptual model, inputs
and outputs
of
MainController.ock unit are given in
Table 1.
Table 2 presents the BCI control system states
.The first state
1C0 of the proposed BCI control system is the Idle state (
Table 2). It is an idle state in which the connection to the TruScan 24 electroencephalograph is checked. The process of establishing the connection with the electroencephalograph is simulated by the variable eegStatus, which takes the value 1 for a correctly established connection. Idle state activity is represented by the idle variable. After receiving the connection information, the system remains in Idle status until the BCI system is started by the Start button being pressed, which is reflected in the start variable.
When there is no error related to the connection to the TruScan 24 device and the Start button is on, the system goes into 2C0 Running state, in which the light stimulator is activated first. The activation of the light stimulator is represented by the variable led = 1. In this state there are three sub-states: Waiting 21C0, DataRecStart 22C0, and DataRecStop 23C0. After starting the light stimulator, the system goes to the Waiting sub-state. From this sub-state, the system goes to DataRecStart after 1 s. This sub-state is responsible for the process of switching on the EEG signal recording by the TruScan 24 device, which has been set for 5 s. The DataRecStart subroutine activity is indicated by the variable dataRec. In the next step, after the 5 s have passed, the system goes to the DataRecStop sub-state where the system disables the EEG signal registration process.
After leaving the DataRecStop sub-station, the quality of recorded data is checked. For this purpose, errorLen and errorImp functions are used. The errorLen function examines the length of the recorded signal. The result of the function is error error2 = 1, when the registered signal does not contain an adequate number of samples. The errorImp function examines the impedance value of measuring electrodes. The result of this function is error error3 = 1, when the impedance value exceeds 5 kΩ. If error2 or error3 occurs, immediate return to Idle status takes place. State 2C0 Running is abandoned when the connection to the electroencephalograph is broken error1 = 1 or when the value of start = 0 (start button disabling). The exit from the state turns off the light stimulator (led = 0).
When errors error2 and error3 do not occur, the system goes to DataProcessing state 3C0. The activity of this state is represented by the variable dataProc. This 3C0 state consists of four substates, listed in the order of calling: 31C0 DataShowing, 32C0 DataPreprocessing, 33C0 FeatureExtraction and 34C0 Classification. In the DataShowing sub-state, the recorded results are displayed using the show function. The dataShow value represents the DataShowing sub-state activity. Then the system goes to DataPreprocessing sub-state 32C0, in which the pre-processing of the registered signal is performed, signaled by the variable dataPre = 1.
Pre-processing of the registered signal consists in extracting a fragment of data of a specific length (dataPreProc function) and filtration using the predefined filters (filtration function). The obtained results are further analyzed in the FeatureExtraction state 33C0. The purpose of this state is to isolate characteristic features from the signal.
In order to test the designed BCI state machine, the detection of characteristic parameters was based on the FFT analysis [
15]. This method is based on reading the frequency at which the largest amplitude value was obtained. Determination of the Fourier transform in the FeatureExtraction state
33C0 is performed by the fftProc function, and the reading of the frequency with the highest amplitude value is performed by the fMaxFind function. The FeatureExtraction state activity represents the fEPoc variable. The last sub-state
34C0 of DataProcessing
3C0 state is the Classification, in which a classification of characteristics and sending a control command to the mobile robot is made. Since the task of the developed BCI system is the mobile robot control, one of five characteristics (modes) may be selected as a result of classification:
- -
MoveForward;
- -
MoveRight;
- -
MoveBack;
- -
MoveLeft;
- -
Unclassified.
The command controlling the robot movement is called after that in the state machine using the coder. extrinsic function.