# An Assistive Technology for Braille Users to Support Mathematical Learning: A Semantic Retrieval System

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. State of the Art

#### 2.1. Accessibility to Mathematical Content by People with Visual Impairments and Blindness

#### 2.1.1. Accessibility

- Screen readers are software applications that allow people with visual impairments to listen to or read the content of a text. It converts the text displayed on the computer screen into speech synthesizer or braille, and the former requires just a simple audio output. However, braille transcription requires additional hardware called refreshable braille displays. In the literature, there are many software solutions, like JAWS (Job Access With Speech), NVDA (Nonvisual Desktop Access), VoiceOver, and Windows Eyes [8].
- A refreshable braille display is an electromechanical system used by people with blindness to read with their fingers like braille hardcopy to decipher the information displayed on the screen. This device can display up to 80 characters on the screen into ephemeral pins, and it changes continuously when the user moves the cursor via control keys. This hardware can also send commands to the computer through the function keys and the routing cursor. With this feature, the blind can perform research, takes notes, and communicate with other people. This device can be used simultaneously or along with a speech-synthesizer system.
- Scanner and Optical Character Recognition allows visually impaired people to scan and read printed books. By connecting this hardware with the computer, the printed document can be scanned and converted into an electronic file that is displayed as text on the computer screen. This text can be read and/or listened to by the blind using refreshable braille displays and screen reader software, respectively. The disadvantage of this material lies in the limitation of digitizing all types of information like images, mathematical notations, and tables.

#### 2.1.2. Assistive Technologies for Braille Users to Study and Access Mathematical Content

#### 2.2. Retrieval of Mathematical-Expression Systems

## 3. Mathematical Braille

#### 3.1. Mathematical Braille Codes

- Mathematical braille notation, used in France.
- Unified Mathematical Code (UMC), practiced in both Latin America and Spain.
- The Nemeth Code, developed by the mathematician Abraham Nemeth, primarily used in the United States and the Middle East.
- The Marburg Code, used in Germany and Austria.
- Mathematical Braille Code, created by the authority of the United Kingdom and used in the United Kingdom and Ireland.

#### 3.2. Mathematical Braille Converters

- Labradoor: (LaTeX-to-braille-door) It converts a LaTeX document including mathematical formulae as well as literary text in braille notation. This project is one of the first attempts at automating the conversion of standard-form mathematical text to print the works in braille, and it is used to produce mathematical texts for secondary-education and university students. Labrador uses only Marburg Braille notation and there is no conversion with other national codes.
- Liblouis [11]: It is capable of transcribing numerous document formats (such as DTBook and XML) in braille and supports numerous languages. Liblouis also supports mathematical braille code like Nemeth and Marburg. It is developed as a library to be used in other programs.
- BraMaNet [34]: Software that converts Presentation MathML only into French Braille. The transcribed file is a standard textual file so that it can easily be used by any other braille transcription software, such as BraMaNetest based on XSLT technology. Diverse parameters are authorized, including the possibility of modifying the braille output table in order to adapt itself to any material. It also comes with a script called MetaBraMaNet that automatically converts from a Microsoft Word document containing mathematical formulae written in MathType to braille.
- UMCL [13]: (Universal Maths Conversion Library) It is a software project for the conversion of mathematical formulae between various formats: MathML specific notation and braille.
- Infty [35]: This is an integrated system aiming to make printed scientific documents, including mathematical formulae, accessible. The system consists of three applications’ components: an OCR system named “InftyReader”, a publisher named “InftyEditor”, and conversion tools in various formats. The vocal interface ChattyInfty [36] allows partially sighted users to access and publish the expressions with a speech output.
- Math2Braille: open-source software that allows the conversion of MathML files into braille. The process of converting MathML into braille relies on protocols and procedures that were developed in a previous project on access to music.

## 4. Retrieval Mathematical Equations System for People with Visual Impairments and Blindness

#### 4.1. Transcription of Braille Expression into MathML Code

#### 4.2. Semantic-Tree Construction

- One-dimensional: Symbols are horizontally connected, e.g., expressions connected by operators ‘+’, ‘–‘, ‘*’, etc. Symbols of this type of relation are connected by the same horizontal label, <mrow>, in MathML.
- Bidimensional: Symbols are connected in nonlinear relations. For example, √, ∑, ∫, ∏, etc.

- For example:
- $\sqrt{x+1}+{x}^{2}ln\left({x}^{2}+1\right)$
- Becomes
- $\sqrt{exp}+exp\text{}ln\left(exp\right)$

#### 4.3. Feature Extraction

- E:
- Database set of the equations E1, E2, E3, …, En.
- T:
- Set of features terms of a database’s equations. The terms are the feature-vector variables: operators (‘+’, ‘–’, ‘*’ …) and functions (cos, sin, sqrt, log, …).
- V(T):
- Space vector of dimension $\left|T\right|=n\in \mathbb{N}$ is defined as the number of terms contained in universe T.
- V
_{il}: - Features vector of equation Ei in an l level of the multilevel tree.
- W
_{ikl}: - Value of the relevance weight of feature term T
_{k}in equation Ei in the l level of the multilevel tree. In our case, if T_{k}does not belong to the descriptor terms of equation Eil, W_{ikl}takes a value of 0; otherwise, W_{ikl}takes a value of 1.

_{i}(V

_{i}

_{1}, V

_{i}

_{2}, …, V

_{in}). The first level’s vector V

_{i1}is obtained by calculating the occurrence number of each term T

_{k}of vector space V(T) in equation Ei while ignoring the content of the terms “exp” and stocking them in features vector V

_{il}. The “exp” terms are treated by repeating the same method applied at the first level. We repeat the same procedure for all remaining levels of equation Ei. The variable and constant terms contain the number of occurrences of the different variables and constants that exist in each level in equation Ei.

#### 4.4. Classification

_{Q}, where V

_{Q}(V

_{Q}

_{1}, V

_{Q}

_{2}, …, V

_{Qn}). The search for similarities between query and database equations is performed by the distance calculation. Therefore, we proposed in this paper to use the K-Nearest-Neighbor (KNN) method.

_{Q}where V

_{Q}(V

_{Q}

_{1}, V

_{Q}

_{2}, …, V

_{Qn}) stemming from the extraction method. The distance formulae can be expressed as follows:

_{Ql}and V

_{yl}are the features vectors of the request equation and an equation E

_{y}, respectively, for level l.

_{Q}(V

_{Q}

_{1}, V

_{Q}

_{2}, …, V

_{Qn}), as a first step, we search all mathematical expressions having similar features vectors of the first level V

_{i}

_{1}to V

_{Q}

_{1}by the calculation of the distance between them. Then, we proceed to the next level just for the similar equations retrieved in the first level. This procedure is repeated in the second level. We continue with the same technique for all levels of the query until we obtain the vectors closest to the request. Reverse indexing allows us to retrieve all mathematical equations corresponding to similar founded feature vectors.

#### 4.5. Braille Transcription

## 5. Experimental Results

#### 5.1. Database

#### 5.2. System Results

#### 5.3. Discussions

- ⠇⠝⠰⠆⠦⠎⠊⠝⠰⠆⠦⠡⠖⠭⠴⠴ in readable format $ln\left(\mathrm{sin}\left(1+x\right)\right)$
- ⠑⠈⠰⠦⠉⠕⠎⠰⠆⠦⠽⠴⠖⠎⠊⠝⠰⠆⠦⠽⠴⠴ in readable format ${e}^{\left(\mathrm{cos}\left(y\right)+\mathrm{sin}\left(y\right)\right)}$

## 6. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Maćkowski, M.S.; Brzoza, P.F.; Spinczyk, D.R. Tutoring math platform accessible for visually impaired people. Int. J. Comput. Biol. Med.
**2018**, 95, 298–306. [Google Scholar] [CrossRef] [PubMed] - Beck-Winchatz, B.; Riccobono, M.A. Advancing participation of blind students in Science, Technology, Engineering, and Math. Int. J. Adv. Space Res.
**2008**, 42, 1855–1858. [Google Scholar] [CrossRef] - Nemeth, A. The Nemeth Braille Code for Mathematics and Science Notation 1972 Revision; American Printing House for the Blind: Louisville, KY, USA, 1972. [Google Scholar]
- RNIB. Braille Mathematics Notation 1987; Royal National Institute for the Blind: Peterborough, UK, 1987. [Google Scholar]
- Waud, D. What Is Next? Available online: http://www.tex.uk/cgi-bin/texfaq2html (accessed on 5 May 2018).
- Ausbrooks, R.; Buswell, S.; Dalmas, S.; Devitt, S.; Diaz, A.; Hunter, R.; Smith, B.; Soiffer, N.; Sutor, R.; Watt, S. Mathematical Markup Language. (MathML) version 2.0. 2000. Available online: https://www.w3.org/Math/ (accessed on 23 October 2018).
- Asebriy, Z.; Raghay, S.; Bencharef, O.; Kaloun, S. A semantic approach for mathematical expression retrieval. IJACSA
**2016**, 7, 190–194. [Google Scholar] [CrossRef] - Wongkia, W.; Naruedomkul, K.; Cercone, N. i-Math: Automatic math reader for Thai blind and visually impaired students. Int. J. Comput. Math. Appl.
**2012**, 64, 2128–2140. [Google Scholar] [CrossRef] [Green Version] - Vlad, P.C.; Tanja, S.; Lars, B.C. Braille Math Extension to RoboBraille: A Universal Software Solution for Converting Math into Braille. In Proceedings of the ICCHP 2016, Linz, Austria, 13–15 July 2016; pp. 15–18. [Google Scholar]
- Christensen, L.B.; Klaus, J. Multilingual two-way braille translation. In Proceedings of the ICCHP ‘96 Proceedings of the 5th International Conference on Computers Helping People with Special Needs, Linz, Austria, 17–19 July 1996. [Google Scholar]
- LibLouis. Available online: http://www.liblouis.org (accessed on 1 October 2018).
- fMath. Available online: http://www.fmath.info (accessed on 28 September 2018).
- UMCL. Available online: https://sourceforge.net/projects/umcl (accessed on 23 October 2018).
- Stanley, P.B.; Karshmer, A.I. Translating MathML into Nemeth Braille Code. In Proceedings of the 10th International Conference, ICCP 2006, Linz, Austria, 11–13 July 2006; pp. 1175–1182. [Google Scholar]
- Duxbury Systems. Available online: http://www.duxburysystems.com/ (accessed on 27 September 2018).
- Amaya XML Editor. Available online: https://www.w3.org/Amaya/ (accessed on 20 October 2018).
- Gardner, J.A. The Lean Math Accessible MathML Editor. In Proceedings of the ICCHP 2014, Paris, France, 9–11 July 2014; Part I, LNCS 8547. pp. 580–587. [Google Scholar]
- Soiffer, N. A Study of Speech Versus Braille and Large Print of Mathematical Expressions. In Proceedings of the ICCHP 2016, Linz, Austria, 13–15 July 2016; Part I, LNCS 9758. pp. 59–66. [Google Scholar]
- Soiffer, N. Browser-independent Accessibilité Math. In Proceedings of the W4A 15, Florence, Italy, 18–20 May 2015. [Google Scholar]
- Salamonczyk, A.; Pawlowska, J.B. Translating of MathML Formulas to Polish Text, Example Applications in Teaching the Blind. In Proceedings of the 2nd International Conference on Cybernetics (CYBCONF), Gdynia, Poland, 24–26 June 2015. [Google Scholar]
- Liska, M.; Sojka, P.; Ruzicka, M. Combining text and Formula Queries in Math Information Retrieval. In Proceedings of the 24th ACM International Conference Son Information and Knowledge Management (CIKM 2015), New York, NY, USA, 19–23 October 2015. [Google Scholar]
- Aizawa, A.; Kohlhase, M.; Ounis, I.; Schubotz, M. NTCIR-11 Math-2 Task Overview. In Proceedings of the 11th NTCIR Conference on Evaluation of Information Access Technologies, Tokyo, Japan, 9–12 December 2014; pp. 88–98. [Google Scholar]
- Liska, M.; Sojka, P.; Ruzicka, M. Similarity Search for Mathematics: Masaryk University team at the NTCIR-10 Math Task. In Proceedings of the 10th NTCIR Conference, Tokyo, Japan, 18–21 June 2013. [Google Scholar]
- Hu, X.; Gao, L.; Lin, X.; Baker, J.B. WikiMirs: A Mathematical Information Retrieval System for Wikipedia. In Proceedings of the 13th ACM/IEEE-Cs Joint Conference on Digital Libraries ACM, Indianapolis, IN, USA, 22–26 July 2013; pp. 11–20. [Google Scholar]
- Egomath. Available online: http://egomath.projekty.ms.mff.cuni.cz/ (accessed on 18 June 2018).
- Hambasan, R.; Kohlhase, M.; Prodescu, C. MathWebSearch at NTCIR-11. In Proceedings of the 11th NTCIR Conference, Tokyo, Japan, 9–12 December 2014. [Google Scholar]
- Lin, X.; Gao, L.; Hu, X.; Tang, Z.; Xiao, Y.; Liu, X. A mathematical retrieval system for formulae in layout presentations. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR 14), Gold Coast, QLD, Australia, 6–11 July 2014; pp. 697–706. [Google Scholar]
- Nguyen, T.T.; Cheung, S.H.; Chang, K. A lattice-based approach for mathematical search using Formal Concept Analysis. Int. J. Expert Syst. Appl.
**2012**, 39, 5820–5828. [Google Scholar] [CrossRef] - Carpineto, C.; Romano, G. Order-theoretical ranking. Int. J. Am. So. Inf. Sci.
**2000**, 51, 587–601. [Google Scholar] [CrossRef] [Green Version] - Cheung, K.S.K.; Vogel, D. Complexity reduction in lattice-based information retieval. Int. J. Inf. Retr.
**2005**, 8, 285–299. [Google Scholar] [CrossRef] - Hao, S.; Shi, C.; Niu, Z.; Cao, L. Concept coupling learning for improving concept lattice-based document retrieval. Int. J. Eng. Appl. Artif. Intell.
**2018**, 69, 65–75. [Google Scholar] [CrossRef] - Yang, S.Q.; Tian, X.D.; Yu, B.T. A maintenance algorithm of FDS based mathematical expression index. In Proceedings of the International Conference on machine learning and Cybernetics, Lanzhou, China, 13–16 July 2014. [Google Scholar]
- Schellenberg, T.; Yuan, B.; Zanibbi, R. Layout-Based Substitution Tree Indexing and Retrieval for Mathematical Expressions. In Proceedings of the Document Recognition and Retrieval XIX, Burlingame, CA, USA, 22–26 January 2012. [Google Scholar]
- BraMaNet. Available online: http://handy.univ-lyon1.fr/MH/bramanet/bramanet.php (accessed on 24 June 2018).
- Suzuki, M.; Kanahori, T.; Ohtake, N.; Yamaguchi, K. An Integrated OCR Software for Mathematical Documents and Its Output with Accessibility. In Proceedings of the International Conference on Computers for Handicapped Persons, Paris, France, 7–9 July 2004; Springer: Berlin/Heidelberg, Germany, 2004; pp. 648–655. [Google Scholar]
- Komada, T.; Yamaguchi, K.; Kawane, F.; Suzuki, M. New Environment for Visually Disabled Students to Access Scientific Information by Combining Speech Interface and Tactile Graphics. In Proceedings of the International Conference on Computers for Handicapped Persons, Linz, Austria, 11–13 July 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 1183–1190. [Google Scholar]
- Duda, R.O.; Hart, P.E.; Stork, D.G. Pattern Classification, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, 2000; p. 680. [Google Scholar]
- Accessible Equation Editor. Available online: http://accessibility.pearson.com/aee/ (accessed on 7 July 2018).
- Maillo, J.; Ramírez, S.; Herrera, I.T.F. kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data. Int. J. Knowl.-Based Syst.
**2017**, 117, 3–15. [Google Scholar] [CrossRef] [Green Version] - Pedronnette, D.C.G.; Gonçalves, F.M.F.; Guilherme, I.R. Unsupervised manifold learning through reciprocal kNN graph and Connected Components for image retrieval tasks. Int. J. Pattern Recogn.
**2018**, 75, 161–174. [Google Scholar] [CrossRef] - Zhan, Y.; Liu, J.; Gou, J.; Wang, M. A video semantic detection method based on locality-sensitive discriminant sparse representation and weighted KNN. Int. J. Visual Commun. Image Represent.
**2016**, 41, 65–73. [Google Scholar] [CrossRef]

**Figure 6.**Multilevel tree construction of equation $\mathrm{cos}\left(x\right)\ast sin\left(ln\left(x\right)\right)$.

**Table 1.**Differences between mathematical braille codes: French notation, Unified Mathematical Code (UMC) and Nemeth.

Symbols | French | UMC | Nemeth |
---|---|---|---|

1 | |||

2 | |||

3 | |||

4 | |||

5 | |||

6 | |||

7 | |||

8 | |||

9 | |||

0 | |||

+ | |||

- | |||

* | |||

/ | |||

^ | |||

$\surd $ | |||

= | |||

( | |||

) | |||

[ | |||

] | |||

{ | |||

} |

Queries | Queries in Braille | Results | ||
---|---|---|---|---|

$ln\left(x\right)$ | ⠇⠝⠦⠭⠴ | ⠇⠝⠦⠭⠴⠖⠭ | ⠇⠝⠦⠁⠖⠃⠴ | ⠇⠝⠦⠽⠴ |

$ln\left(x\right)+x$ | $ln\left(a+b\right)$ | $ln\left(y\right)$ | ||

⠇⠝⠦⠭⠴⠖⠻ | ⠇⠝⠦⠡⠤⠭⠴ | ⠡⠤⠇⠝ ⠦⠜⠁⠖⠣⠴ | ||

$ln\left(x\right)+7$ | $ln\left(1-x\right)$ | $1-ln\left(\sqrt{a}+2\right)$ | ||

$sin\left(x\right)$ | ⠎⠊⠝⠦⠭⠴ | ⠎⠊⠝⠦⠭⠴ | ⠭⠎⠊⠝⠦⠭⠴ | ⠎⠊⠝⠦⠱⠖⠭⠴ |

$sin\left(x\right)$ | $xsin\left(x\right)$ | $sin\left(5+x\right)$ | ||

⠎⠊⠝⠦⠣⠴⠖⠎⠊⠝ ⠦⠡⠖⠃⠴ | ⠯⠎⠊⠝ ⠦⠭⠖⠣⠴⠙⠭ | ⠎⠊⠝⠦⠱⠖⠭⠈⠣⠴ | ||

$\mathrm{sin}\left(2\right)+sin\left(1+b\right)$ | ${{\displaystyle \int}}^{}sin\left(x+2\right)dx$ | $sin\left(5+{x}^{2}\right)$ | ||

$1+x$ | ⠡⠖⠭ | ⠡⠖⠭⠽ | ⠵⠀⠭⠖⠱ | ⠡⠌⠭⠖⠽ |

$1+xy$ | $zx+5$ | $\frac{1}{x}+y$ | ||

⠡⠌⠰⠭⠖⠡⠆ | ⠑⠦⠣⠴⠖⠡ | ⠑⠦⠁⠖⠁⠌⠣⠴ | ||

$\frac{1}{x+1}$ | $e\left(2\right)+1$ | $e\left(a+\frac{a}{2}\right)$ | ||

$\frac{1}{x}$ | ⠡⠌⠭ | ⠡⠌⠭⠖⠽ | ⠡⠌⠰⠭⠖⠡⠆ | ⠰⠡⠖⠭⠆⠌⠰ ⠞⠁⠝⠦⠭⠴⠆ |

$\frac{1}{x}+y$ | $\frac{1}{x+1}$ | $\frac{1+x}{tan\left(x\right)}$ | ||

⠰⠣⠀⠭⠆⠌⠰⠡⠤⠭⠆ | ⠰⠭⠖⠡⠆⠌⠰⠭⠤⠡⠆ | ⠰⠇⠝⠦⠭⠴⠖⠡⠆⠌⠭ | ||

$\frac{2x}{1-x}$ | $\frac{x+1}{x-1}$ | $\frac{ln\left(x\right)+1}{x}$ | ||

$ln\left(\mathrm{sin}\left(1+x\right)\right)$ | ⠇⠝⠰⠀⠆⠦⠎⠊⠝⠰ ⠆⠦⠡⠖⠭⠴⠴ | ⠣⠇⠝⠦⠭⠴⠇⠝⠦⠽⠴ ⠶⠡ | ⠡⠤⠇⠝⠦⠜⠁⠖⠣⠴ | ⠇⠝⠦⠁⠴⠤⠇⠝ ⠦⠁⠖⠃⠴ |

$2ln\left(x\right)ln\left(y\right)=1$ | $1-ln\left(\sqrt{a}+2\right)$ | $ln\left(a\right)-ln\left(a+b\right)$ | ||

⠭⠀⠎⠊⠝⠦⠭⠴ | ⠇⠝⠦⠡⠤⠭⠴ | ⠇⠝⠦⠭⠴⠖⠇⠝ ⠦⠭⠖⠽⠴ | ||

$xsin\left(x\right)$ | $ln\left(1-x\right)$ | $ln\left(x\right)+ln\left(x+y\right)$ | ||

${e}^{\left(\mathrm{cos}\left(y\right)+\mathrm{sin}\left(y\right)\right)}$ | ⠑⠈⠰⠦⠉⠕⠎⠰⠀⠆⠦⠽⠴ ⠖⠎⠊⠝⠰⠀⠆⠦⠽⠴⠴ | ⠰⠡⠖⠎⠊⠝⠦⠭⠴ ⠆⠌⠰⠡⠤⠭⠆ | ⠑⠦⠣⠴⠤⠡ | ⠰⠡⠤⠉⠕⠎⠦⠭⠴ ⠆⠌⠰⠡⠤⠭⠆ |

$\frac{1+sin\left(x\right)}{1-x}$ | $e\left(2\right)-1$ | $\frac{1-cos\left(x\right)}{1-x}$ | ||

⠑⠦⠣⠴⠖⠡ | ⠯⠎⠊⠝⠦⠭⠖⠣⠴⠙⠭ | ⠎⠊⠝⠦⠑⠦⠭⠖⠡⠴⠴ | ||

$e\left(2\right)+1$ | ${{\displaystyle \int}}^{}sin\left(x+2\right)dx$ | $sin\left(e\left(x+1\right)\right)$ |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Asebriy, Z.; Raghay, S.; Bencharef, O.
An Assistive Technology for Braille Users to Support Mathematical Learning: A Semantic Retrieval System. *Symmetry* **2018**, *10*, 547.
https://doi.org/10.3390/sym10110547

**AMA Style**

Asebriy Z, Raghay S, Bencharef O.
An Assistive Technology for Braille Users to Support Mathematical Learning: A Semantic Retrieval System. *Symmetry*. 2018; 10(11):547.
https://doi.org/10.3390/sym10110547

**Chicago/Turabian Style**

Asebriy, Zahra, Said Raghay, and Omar Bencharef.
2018. "An Assistive Technology for Braille Users to Support Mathematical Learning: A Semantic Retrieval System" *Symmetry* 10, no. 11: 547.
https://doi.org/10.3390/sym10110547