MuPAD - Dynamic Modules [1]

How to MuPAD scanners with flex

Andreas Sorgatz (andi@uni-paderborn.de) - 01. Dec. 1996

lex (or the GNU version flex) is a UNIX standard tool to create scanners for lexicographical analysis of text input files. The scanner generator takes a file 'demo.l' which describes the scanner by utilizing regular expressions as input.
[About] -- [Demo] -- [Help Pages] -- [Specification] -- [Creation] -- [Availability]

About flex

The program flex is available for all UNIX systems and should be installed on your local system. However, you can get it via anonymous ftp from any GNU mirror site. For detailed information refer to the manual flex(1).

The MuPAD flex interface was tested on the following plattforms (further ports are planned).

PC/AT (Linux 2.0.7),
SunSPARC (Solaris 2.5).

A Demo Scanner

Figure 1 demonstrates how the MuPAD scanner module can be used. After the scanner demo is loaded the function demo::open is used to specify and open the input (text) file 'demo.inp'. The function demo::token scans the file and returns the next analyzed token. Tokens are identified by non-negative numbers, whereby the token 0 marks the EOF (end-of-file) token.

Fig. 1: How To use A MuPAD Scanner Module

module(demo):                                   # load the scanner module   #
demo::open("demo.inp"):                         # open the input text file  #

while( (t:=demo::token()) > 0 ) do              # scan until EOF is reached #
   print( Unquoted,                             
          " line= ". expr2text(demo::line()).
          " token= ".expr2text(t).
          " text= ". demo::text() 
   );
end_while:

demo::close():                                  # close the input text file #

The text of the current token is delivered by the function demo::text, whereas the function demo::line returns the current line number. It is also possible to put back a token by using the function demo::back in order to rescan it.

The Online Help Pages

The following help pages are placed in the file demo.mdh in the same directory as the scanner module demo.mdm itself. They can be read during a MuPAD session via the function demo::doc() - after the module demo was loaded.

Fig. 2: The Online Help Pages

MODULE
    demo - A MuPAD module interface to a (f)lex scanner

INTRODUCTION
    This module defines a MuPAD interface to a scanner created with the
    scanner generator (f)lex. The function token() returns non-negative
    integers, with EOFILE=0. The string of the current token can be ex-
    tracted with text(). back() allows the user to put back the current
    token, to be re-scanned with the next call of the function token().

    For technical information please refer to the manual of flex(1) and 
    also to the source file of this module.   MuPAD scanner modules are 
    static modules, that will not be unloaded automatically.   They can
    be created with the UNIX shell script 'mkscan',  which is available 
    in the contribution directory of official MuPAD ftp sites.

INTERFACE
    doc, back, close, line, open, text, token


NAME demo::doc - Online documentation SYNOPSIS demo::doc() demo::doc( func ) ARGUMENTS func - Function name without the prefix "demo::" (DOM_STRING) DESCRIPTION Displays a brief description of this scanner module , respectively the function demo::'func'. EXAMPLE: >> demo::doc("token"); [...] SEE ALSO module::help
NAME demo::back - Puts back the current token, to be re-scanned later SYNOPSIS demo::back() ARGUMENTS - DESCRIPTION Puts back the current token, to be re-scanned with the next call of the function token(). back() cannot be called again before the put back token was re-scanned by token(). The function returns TRUE if the token was put back and FALSE if not. NOTE: After back() was called, text() returns the empty character string ("") until the token is re-scanned by token(). EXAMPLE: >> demo::token(); 13 >> demo::back(); TRUE >> demo::token(), demo::token(); 13, 7 SEE ALSO demo::token
NAME demo::close - Closes the current input file SYNOPSIS demo::close() ARGUMENTS - DESCRIPTION Closes the current input file of this scanner module. The function returns the value DOM_NULL. EXAMPLE: >> demo::open("test"); TRUE >> demo::close(); SEE ALSO demo::open
NAME demo::line - Returns the current line number. SYNOPSIS demo::line() ARGUMENTS - DESCRIPTION Returns the current line number. The line numbers starts with 1. EXAMPLE: >> demo::open("test"); TRUE >> demo::line(); 1 SEE ALSO demo::text
NAME demo::open - Opens a new scanner input file SYNOPSIS demo::open( file ) ARGUMENTS file - A file name (DOM_STRING) DESCRIPTION Opens a new scanner input file. If there is an open scanner file of this scanner module, then it is closed before. The function returns TRUE if the "file" has been opened and FALSE otherwise. EXAMPLE: >> demo::open("test"); TRUE >> demo::token(); 2 SEE ALSO demo::close, demo::token
NAME demo::text - Returns the character string of the current token SYNOPSIS demo::text() ARGUMENTS - DESCRIPTION Returns the character string of the current token. Before the first token is read as well as at the end of the input file this function returns the empty character string (""). This is also be done when back() was called and the token was not re-scanned yet. EXAMPLE: >> demo::text(); "42" >> demo::token(); 0 >> demo::text(); "" SEE ALSO demo::line, demo::token
NAME demo::token - Returns the next token read from the input file SYNOPSIS demo::token() ARGUMENTS - DESCRIPTION Returns the next token read from the current input file. Tokens are non-negative integers with EOFILE=0, UNKNOWN=1, ... EXAMPLE: >> demo::token(); 2 >> demo::text(); "while" >> demo::token(); 0 SEE ALSO demo::line, demo::text

How To Create A Scanner Module

The Scanner Specification

To create MuPAD scanner modules the shell script mkscan can be used. This shell script takes a scanner specification file 'x.l' as input and creates an executable scanner module 'x.mdm' as well as a help file .mdh' as output. Both are placed in the current working directory. A number of additional, temporary files are created during the compilation but they are not needed after the module is compiled. Figure 3 shows the flex specification file of the demo scanner.

Fig. 3: Creating Scanner Module magnum

/******************************************************************************/
/* FILE   : demo.l - A sample of a scanner description file for (f)lex        */
/* AUTHOR : Andreas Sorgatz (andi@uni-paderborn.de)                           */
/* CREATED: 24. Nov. 1996                                                     */
/* CHANGED: 26. Nov. 1996                                                     */
/******************************************************************************/

/** Definiton of tokens:  EOFILE=0 and UNKNOWN=1 must allways be first. All ***/
/** tokens are  non-negative integer. They will be returned by the function ***/
/** 'scan::token()'. The rule for UNKNOWN tokens is defined with the module ***/
/** source code (refer to 'ECHO' and the manual of '(f)lex'                 ***/
/**/
%{
    enum { EOFILE=0, UNKNOWN=1, NEWLINE, COMMENT, IDENT, INT, FLOAT, STRING };
%}

/** Definition of regular expressions *****************************************/
/**/
wspace      [\t ]
nline       [\n]
Ccomment    \/\/[^\n]*
comment     {Ccomment}

digit       [0-9]
letter      [A-Za-z_]

int         {digit}+
float       {digit}+(\.{digit}+)?((E|e)[+-]?{digit}+)?
ident       {letter}({letter}|{digit})*
string      \"[^\"]*\"

/** The rules: the variable 'ScanLine' is defined in the module source code ***/
/**/
%%
{wspace}+   { }                                    /* Eat up all whitespaces */
{nline}     { ++ScanLine; }                        /* Eat up all linefeeds   */
{comment}   { }                                    /* Eat up all comments    */

{ident}     { return( IDENT    ); }
{int}       { return( INT      ); }
{float}     { return( FLOAT    ); }
{string}    { return( STRING   ); }

%%
/** User code, this section may contain help routines *************************/

This seems to be a bit complicated, but is easy to handle in most cases. Remember, all you have to do is to define regular expression. However, for more complex languages you can also use advanced features of flex(1).

The Shell Script mkscan

To create the final scanner module all you have to do is to call the shell script mkscan with the argument demo. Both the creation as well as the result of the MuPAD commands listed in figure 1 is shown in figure 4.

Fig. 4: Creation and Usage of demo

andi> ./mkscan demo
## Creating (f)lex scanner source code #############################
##
demo.l ==> demo.yy.c
## Creating the module source code #################################
##
mkscan ==> demo.C
## Creating the module help file ###################################
##
mkscan ==> demo.mdh
## Compiling the dynamic scanner module ############################
##
demo.l, demo.C ==> demo.mdm

MMG -- MuPAD-Module-Generator -- V-1.3.0  Oct.96
Mesg.: Scanning source file ...
Mesg.: 6 function(s) and 1 option(s) found in 1665 lines
Mesg.: Creating  extended module code ...
Mesg.: Compiling extended module code ...

g++ -fpic  -c MMGdemo.C -o demo.o 
    -DPARI_C_PLUSPLUS -DLONG_IS_32BIT -DPOINTER_IS_32BIT 
    -I/home/andi/MuPAD/DEV/share/mmg/include/kernel 
    -I/home/andi/MuPAD/DEV/share/mmg/include/pari 
    -I/home/andi/MuPAD/DEV/share/mmg/include/mmt

Mesg.: Linking dynamic module ...

ld -shared -o demo.mdm demo.o 
    -L/usr/local/gnu/lib -L/usr/local/lib -lfl 
    -L/home/andi/MuPAD/DEV/i386/lib

Mesg.: Ok


# Some ouput of the MuPAD commands listed in figure 1 # line= 7 token= 4 text= dynamic line= 7 token= 4 text= modules line= 7 token= 4 text= are line= 7 token= 4 text= useful line= 11 token= 5 text= 7 line= 11 token= 5 text= 13 line= 11 token= 5 text= 42 line= 11 token= 5 text= 666 line= 15 token= 6 text= 1.2 line= 15 token= 6 text= 1.0 line= 15 token= 6 text= 1e-2 line= 15 token= 6 text= 1.0e-2 line= 19 token= 7 text= "These" line= 19 token= 7 text= "are" line= 19 token= 7 text= "four" line= 19 token= 7 text= "strings" line= 23 token= 1 text= - # 1 = unknown token # line= 23 token= 1 text= + line= 23 token= 1 text= * line= 23 token= 1 text= /

Appendix

The shell script mkscan for the automatic creation of MuPAD scanner modules will be available with the MuPAD release 1.3. Please refer to the contrib directory of any official MuPAD ftp site.

Bibliography

[1] Dynamische Module, A. Sorgatz, MuPAD Reports, Teubner Verlag, Stuttgart, Oktober 1996. http://math-www.uni-paderborn.de/MuPAD/BIB/sorgatz96.html
[2] A first integration of the factorization function of Magnum in MuPAD 1.2.9a was done by Paul Zimmermann: http://www.loria.fr/~zimmerma/mupad/magnum.html.
[3] Integration of Magnum in MuPAD
[4] Using the NAGC Library in MuPAD


Author: Andreas Sorgatz (andi@uni-paderborn.de)
Last update: 16. Dec. 1996