Constrained Dynamic Lexer - Revision history

imported>Pascal at 09:10, 1 March 2010

2010-03-01T09:10:36Z

← Older revision		Revision as of 09:10, 1 March 2010
Line 35:		Line 35:

	The generic parser will hold the current construction of identifiers.		The generic parser will hold the current construction of identifiers.
	In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule :		In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule:

	<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*		<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*

	and implemented with the following rules :		and implemented with the following rules:

	<tt><identifier_start></tt> can be any character valid under the terms of <tt>Character.isJavaIdentifierStart(char ch)</tt> except :		<tt><identifier_start></tt> can be any character valid under the terms of <tt>Character.isJavaIdentifierStart(char ch)</tt> except:
	* '$' which indicates the beginning of a predicate variable's name,		* '$' which indicates the beginning of a predicate variable's name,
	* 'λ' (\u03bb)		* 'λ' (\u03bb)

	<tt><identifier_extend></tt> can be any character valid under the terms of <tt>Character.isJavaIdentifierPart(char ch)</tt> except :		<tt><identifier_extend></tt> can be any character valid under the terms of <tt>Character.isJavaIdentifierPart(char ch)</tt> except:
	* '$'		* '$'
	* 'λ' (\u03bb)		* 'λ' (\u03bb)
Line 51:		Line 51:

	=== Open Questions ===		=== Open Questions ===
	There are currently 4 types of lexems : identifiers, numbers, predicate variables, and keywords.		There are currently 4 types of lexems: identifiers, numbers, predicate variables, and keywords.

	Is it valuable to complement those 4 types of lexems including comments that could this way be placed anywhere?		Is it valuable to complement those 4 types of lexems including comments that could this way be placed anywhere?

imported>Pascal: /* Identifiers */

2010-03-01T09:08:55Z

Identifiers

← Older revision		Revision as of 09:08, 1 March 2010
Line 37:		Line 37:
	In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule :		In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule :

	<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*		<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*

	and implemented with the following rules :		and implemented with the following rules :

	<identifier_start> can be any character valid under the terms of Character.isJavaIdentifierStart(char ch) except :		<tt><identifier_start></tt> can be any character valid under the terms of <tt>Character.isJavaIdentifierStart(char ch)</tt> except :
	* '$' which indicates the beginning of a predicate variable's name,		* '$' which indicates the beginning of a predicate variable's name,
	* 'λ' (\u03bb)		* 'λ' (\u03bb)

	<identifier_extend> can be any character valid under the terms of Character.isJavaIdentifierPart(char ch) except :		<tt><identifier_extend></tt> can be any character valid under the terms of <tt>Character.isJavaIdentifierPart(char ch)</tt> except :
	* '$'		* '$'
	* 'λ' (\u03bb)		* 'λ' (\u03bb)

imported>Tommy: /* Identifiers */

2010-02-25T18:18:18Z

Identifiers

← Older revision		Revision as of 18:18, 25 February 2010
Line 43:		Line 43:
	<identifier_start> can be any character valid under the terms of Character.isJavaIdentifierStart(char ch) except :		<identifier_start> can be any character valid under the terms of Character.isJavaIdentifierStart(char ch) except :
	* '$' which indicates the beginning of a predicate variable's name,		* '$' which indicates the beginning of a predicate variable's name,
	* 'λ' (\~~u0033~~)		* 'λ' (\u03bb)

	<identifier_extend> can be any character valid under the terms of Character.isJavaIdentifierPart(char ch) except :		<identifier_extend> can be any character valid under the terms of Character.isJavaIdentifierPart(char ch) except :
	* '$'		* '$'
	* 'λ' (\~~u0033~~)		* 'λ' (\u03bb)
	and possibly completed by a "'".		and possibly completed by a "'".

imported>Tommy: /* Identifiers */

2010-02-25T17:49:36Z

Identifiers

← Older revision		Revision as of 17:49, 25 February 2010
Line 40:		Line 40:

	and implemented with the following rules :		and implemented with the following rules :

	<identifier_start> can be any character valid under the terms of Character.isJavaIdentifierStart(char ch) except :		<identifier_start> can be any character valid under the terms of Character.isJavaIdentifierStart(char ch) except :
	* '$' which indicates the beginning of a predicate variable's name,		* '$' which indicates the beginning of a predicate variable's name,

imported>Tommy: /* Identifiers */

2010-02-25T17:48:58Z

Identifiers

← Older revision		Revision as of 17:48, 25 February 2010
Line 36:		Line 36:
	The generic parser will hold the current construction of identifiers.		The generic parser will hold the current construction of identifiers.
	In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule :		In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule :

	<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*		<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*

Line 47:		Line 48:
	* 'λ' (\u0033)		* 'λ' (\u0033)
	and possibly completed by a "'".		and possibly completed by a "'".


	=== Open Questions ===		=== Open Questions ===

imported>Tommy: Added informations about identifiers

2010-02-25T17:48:28Z

Added informations about identifiers

← Older revision		Revision as of 17:48, 25 February 2010
Line 29:		Line 29:

	[[Image:RecognitionGraph.png\|center\|A sample recognition graph]]		[[Image:RecognitionGraph.png\|center\|A sample recognition graph]]

			== Same behaviour ==

			=== Identifiers ===

			The generic parser will hold the current construction of identifiers.
			In the current version of the [[Event-B Mathematical Language]], identifiers are defined against this Unicode syntactical rule :
			<identifier> := <identifier_start> (<identifier_start> \| <identifier_extend>)*

			and implemented with the following rules :
			<identifier_start> can be any character valid under the terms of Character.isJavaIdentifierStart(char ch) except :
			* '$' which indicates the beginning of a predicate variable's name,
			* 'λ' (\u0033)

			<identifier_extend> can be any character valid under the terms of Character.isJavaIdentifierPart(char ch) except :
			* '$'
			* 'λ' (\u0033)
			and possibly completed by a "'".


	=== Open Questions ===		=== Open Questions ===

imported>Tommy: /* Open Questions */

2010-02-24T17:50:50Z

Open Questions

← Older revision		Revision as of 17:50, 24 February 2010
Line 32:		Line 32:
	=== Open Questions ===		=== Open Questions ===
	There are currently 4 types of lexems : identifiers, numbers, predicate variables, and keywords.		There are currently 4 types of lexems : identifiers, numbers, predicate variables, and keywords.

	Is it valuable to complement those 4 types of lexems including comments that could this way be placed anywhere?		Is it valuable to complement those 4 types of lexems including comments that could this way be placed anywhere?


	[[Category:Design proposal]]		[[Category:Design proposal]]

imported>Tommy: New page: {{TOCright}} This page describes the requirements for a '''parameterizable lexical analyser''' (i.e. lexer) for the Event-B Mathematical Language. A first design proposal is also dra...

2010-02-24T17:50:27Z

New page: {{TOCright}} This page describes the requirements for a '''parameterizable lexical analyser''' (i.e. lexer) for the Event-B Mathematical Language. A first design proposal is also dra...

New page

{{TOCright}}

This page describes the requirements for a '''parameterizable lexical analyser''' (i.e. lexer) for the [[Event-B Mathematical Language]].

A first design proposal is also drafted.

== Requirements ==
In order to be usable, [[mathematical extensions]] require that the Event-B
mathematical language syntax can be extended by the final user.

The '''lexical analyser''' and the syntactic parser thus have to be extensible in a
simple enough way (from a user point of view).

=== Requirements Exported by the Current Language Design ===
* lexems and tokens to be used are known at lexer's instantiation.

=== Requirements Exported by the Dynamic Feature ===
* lexems and their corresponding tokens are collected from [[Event-B Mathematical Language]]'s set of keywords and from [[mathematical extensions]],
* a given lexem corresponds to a unique token.

== Design Proposal ==

The keywords defining the [[Event-B Mathematical Language]], or the [[mathematical extensions]] used to parameterize the lexer are converted into a set of directed graphs. Each node represents a state in lexem recognition, and each transition represents a choice in the symbol to continue the recognition with.
An array of symbols to be recognized, can be associated with a node in order to group a sequence of transitions where no choice is needed to be done. This is an optimisation to prevent a cumbersome creation of node objects.

When a lexem is scanned, the lexer tries to find an entry node in the set of graphs, that corresponds to its beginning (i.e. its first character). Traversing the graph, the lexer finds out if the current lexem is actually a keyword or not.

Here is a figure describing the recognition graph associated for a lexem beginning with the letter p (where configuration keywords beginning with the letter "p" are : "partition", "prj1", "prj2" and "pred".

[[Image:RecognitionGraph.png|center|A sample recognition graph]]

=== Open Questions ===
There are currently 4 types of lexems : identifiers, numbers, predicate variables, and keywords.
Is it valuable to complement those 4 types of lexems including comments that could this way be placed anywhere?

[[Category:Design proposal]]