Skip to content
GitLab
Commit 05097afd authored by Carsten Rose's avatar Carsten Rose
Browse files

Report: Nesting of level extented to support different types of braces:... 

Report: Nesting of level extented to support different types of braces: '{}[]()<>'. Parsing regexp rewritten to be more precise.
parent fe9592e3
Hide whitespace changes
Inline Side-by-side
extension/Documentation/UsersManual/Index.rst
View file @ 05097afd
......@@ -32,6 +32,7 @@ The title of the QFQ content element will not be rendered. It's only visible in
QFQ Keywords (Bodytext)
^^^^^^^^^^^^^^^^^^^^^^^
+-------------------+---------------------------------------------------------------------------------+
| Name | Explanation |
+===================+=================================================================================+
......@@ -343,10 +344,8 @@ SQL
* The following commands are interpreted as SQL commands:
* SELECT
* INSERT
* UPDATE
* DELETE
* SHOW
* INSERT, UPDATE, DELETE, REPLACE, TRUNCATE
* SHOW, DESCRIBE, EXPLAIN, SET
* A SQL Statement might contain parameters, including additional SQL statements. Inner SQL queries will be executed first.
* All variables will be substituted one by one from inner to outer.
......@@ -923,7 +922,7 @@ current record either to finalize the upload or to delete a previous uploaded fi
* All necessary subdirectories in `fileDestination` are automatically created.
Deleting a record and the referenced file
.........................................
'''''''''''''''''''''''''''''''''''''''''
If the user deletes a record which contains reference(s) to files, such files are deleted too.
......@@ -1141,7 +1140,6 @@ See the example below:
::
10.sql = SELECT id AS _person_id, CONCAT(first_name, " ", last_name, " ") AS name FROM person
10.rsep = <br />
......@@ -1162,15 +1160,16 @@ This would result in
..
Across several lines
^^^^^^^^^^^^^^^^^^^^
Text across several lines
^^^^^^^^^^^^^^^^^^^^^^^^^
To make SQL quieres more readable, it's possible to split a line across several lines. Lines with keywords are on their
own - if a line is not a 'keyword' line, it will be appended at the last keyword line. 'Keyword' lines are detected on:
To make SQL queries, or QFQ records in general, more readable, it's possible to split a line across several lines. Lines
with keywords are on their own (`QFQ Keywords (Bodytext)`_) starts a new line - if a line is not a 'keyword' line, it will
be appended at the last keyword line. 'Keyword' lines are detected on:
* <level>.<keyword> =
* {
* <level> {
* <level>[.<level] {
Example::
......@@ -1189,7 +1188,45 @@ Example::
Nesting of levels
^^^^^^^^^^^^^^^^^
Levels can be nested by using curly brackets. Be carefull to write nothing than whitespaces/newline behind open or closing curly braces::
Levels can be nested. E.g.: ::
10 {
sql = SELECT ...
5 {
sql = SELECT ...
head = ...
}
}
This is equal to: ::
10.sql = SELECT ...
10.5.sql = SELECT ...
10.5.head = ...
By default, curly braces '{}' are used for nesting. Alternatively angle braces '<>', round braces '()' or square
braces '[]' are also possible. To define the braces to use, the **first line** of the bodytext has to be a comment line and the
last character of that line must be one of '{}[]()<>'. The corresponding braces are used for that QFQ record. E.g.: ::
# Specific code. >
10 <
sql = SELECT
head = <script>
data = [
{
10, 20
}
]
</script>
>
Per QFQ tt-content record, only one type of nesting braces can be used.
Be carefull to:
* write nothing else than whitespaces/newline behind an **open brace**
* the **closing brace** has to be alone on a line. ::
10.sql = SELECT 'hello world'
......@@ -1217,6 +1254,9 @@ Levels can be nested by using curly brackets. Be carefull to write nothing than
}
Access to upper column values
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Columns of the upper level result can be accessed via variables, eg. {{10.person_id}} will be replaced by the value in the person_id column.
+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
......@@ -1878,6 +1918,9 @@ Runs batch files or executables on the webserver. In case of an error, returncod
..
Column: _F
^^^^^^^^^^
Challenge 1
'''''''''''
......@@ -2331,8 +2374,8 @@ Same as above, but written in the nested notation::
* Columns starting with a '_' won't be printed but can be accessed as regular columns.
Best practice
=============
Best practice: Form
===================
Debug Report
------------
......@@ -2395,7 +2438,8 @@ unchanged.
::
detail=id:formId,{{SELECT '&', IFNULL(fe.ord,0)+10 FROM Form AS f LEFT JOIN FormElement AS fe ON fe.formId=f.id WHERE f.id={{r:S0}} ORDER BY fe.ord DESC LIMIT 1}}:ord
detail=id:formId,{{SELECT '&', IFNULL(fe.ord,0)+10 FROM Form AS f LEFT JOIN FormElement AS fe ON fe.formId=f.id WHERE
f.id={{r:S0}} ORDER BY fe.ord DESC LIMIT 1}}:ord
* Secondary form, `ord` formelement, field `value`: set
......@@ -2505,3 +2549,61 @@ Relation: `Person.id = Address.personId`
Best Practise: Chart
====================
* QFQ delivers a chart JavaScript lib: https://github.com/nnnick/Chart.js.git. Docs: http://www.chartjs.org/docs/
* The library is not sourced in the HTML page by automatically. To do it, either include the lib
`typo3conf/ext/qfq/Resources/Public/JavaScript/Chart.min.js`:
* in the specific tt_content record (shown below in the example) or
* system wide via Typo3 Template record.
* By splitting HTML and JavaScript code over several lines, take care not accidently to create a 'nesting'-end token. Check
the line after `10.tail =`. It's '}' alone on one line. This is a valid 'nesting'-end token!. There are two options to
circumwait this:
* Do not nest the HTML & JavaScript code - this is not human readable.
* Select '<' / '>' as token (check the first line).
::
# <
10.sql = SELECT '_'
10.head =
<div style="height: 1024px; width: 640px;">
<h3>Distribution of FormElement types over all forms</h3>
<canvas id="barchart" width="1240" height="640"></canvas>
</div>
<script src="typo3conf/ext/qfq/Resources/Public/JavaScript/Chart.min.js"></script>
<script>
$(function () {
var ctx = document.getElementById("barchart");
var barChart = new Chart(ctx, {
type: 'bar',
data: {
10.tail =
}
});
});
</script>
# Labels
10.10 <
sql = SELECT "'", fe.type, "'" FROM FormElement AS fe GROUP BY fe.type ORDER BY fe.type
head = labels: [
tail = ],
rsep = ,
>
# Data
10.20 <
sql = SELECT COUNT(fe.id) FROM FormElement AS fe GROUP BY fe.type ORDER BY fe.type
head = datasets: [ { data: [
tail = ], backgroundColor: "steelblue", label: "FormElements" } ]
rsep = ,
>
extension/qfq/qfq/BodytextParser.php
View file @ 05097afd
......@@ -12,21 +12,27 @@ const NESTING_TOKEN_OPEN = '#&nesting-open-&#';
const NESTING_TOKEN_CLOSE = '#&nesting-close&#';
const NESTING_TOKEN_LENGTH = 17;
class BodytextParser {
class BodytextParser {
/**
* @param $bodytext
* @return mixed
*/
public function process($bodytext) {
$bodytext = $this->trimAndRemoveCommentAndEmptyLine($bodytext);
$nestingOpen = '';
$nestingClose = '';
$bodytext = $this->trimAndRemoveCommentAndEmptyLine($bodytext, $nestingOpen, $nestingClose);
// Encrypt double curly braces to prevent false positives with nesting: form = {{form}}\n
$bodytext = Support::encryptDoubleCurlyBraces($bodytext);
$bodytext = $this->encryptNestingDelimeter($bodytext);
$bodytext = $this->joinLine($bodytext);
$bodytext = $this->unNest($bodytext);
$bodytext = $this->trimAndRemoveCommentAndEmptyLine($bodytext);
$bodytext = $this->joinLine($bodytext, $nestingOpen, $nestingClose);
$bodytext = $this->encryptNestingDelimeter($bodytext, $nestingOpen, $nestingClose);
$bodytext = $this->unNest($bodytext, $nestingOpen, $nestingClose);
$bodytext = $this->trimAndRemoveCommentAndEmptyLine($bodytext, $nestingOpen, $nestingClose);
$bodytext = Support::decryptDoubleCurlyBraces($bodytext);
if (strpos($bodytext, NESTING_TOKEN_OPEN) !== false) {
......@@ -42,54 +48,125 @@ class BodytextParser {
* @return string
*/
private function trimAndRemoveCommentAndEmptyLine($bodytext) {
private function trimAndRemoveCommentAndEmptyLine($bodytext, &$nestingOpen, &$nestingClose) {
$data = array();
$src = explode(PHP_EOL, $bodytext);
if ($src === false) {
return '';
}
$firstLine = trim($src[0]);
foreach ($src as $row) {
$row = trim($row);
if ($row === '' || $row[0] === '#') {
continue;
}
$data[] = $row;
}
$this->setNestingToken($firstLine, $nestingOpen, $nestingClose);
return implode(PHP_EOL, $data);
}
/**
* Encrypt '{\n' and '}\n' by more complex token.
* Set the 'nesting token for this tt-conten record. Valid tokens are {}, <>, [], ().
* If the first line of bodytext is a comment line and the last char of that line is a valid token: set that one.
* If not: set {} as nesting token.
*
* @param $bodytext
* @return mixed
* Example:
* # Some nice text - no token found, take {}
* # ] - []
* # Powefull QFQ: < - <>
*
* @param $firstLine
* @param $nestingOpen
* @param $nestingClose
*/
private function encryptNestingDelimeter($bodytext) {
// Take care that a trailing '}' will be recognised: add '\n'
if (substr($bodytext, -1) === '}') {
$bodytext .= "\n";
private function setNestingToken($firstLine, &$nestingOpen, &$nestingClose) {
if ($nestingOpen !== '') {
return; // tokens already set or not bodytext: do not change.
}
$bodytext = str_replace("{\n", NESTING_TOKEN_OPEN, $bodytext);
$bodytext = str_replace("}\n", NESTING_TOKEN_CLOSE, $bodytext);
return $bodytext;
// Nothing defined: set default {}.
if ($firstLine === false || $firstLine === '' || $firstLine[0] !== '#') {
$nestingOpen = '{';
$nestingClose = '}';
return;
}
$pos = 0;
$tokenList = '{}<>[]()';
// Definition: first line of bodytext, has to be a comment line. If the last char is one of the valid token: set that one.
// Nothing found: set {}.
if ($firstLine[0] === '#') {
$token = substr($firstLine, -1);
$pos = strpos($tokenList, $token);
if ($pos === false) {
$pos = 0;
} else {
if ($pos % 2 === 1) {
$pos -= 1;
}
}
}
$nestingOpen = substr($tokenList, $pos, 1);
$nestingClose = substr($tokenList, $pos + 1, 1);
}
/**
* Join lines, which do not begin with '<level>.<keyword>[ ]='
* Join lines. Preservers Nesting.
*
* Iterates over all lines.
* Is a line a 'new line'?
* no: concat it to the last one.
* yes: flush the buffer, start a new 'new line'
*
* New Line Trigger:
* a: {
* b: }
* c: 20
* d: 20.30
*
* e: 5 {
* f: 5.10 {
*
* g: head =
* h: 10.20.head =
*
* c,d,e,f: ^\d+(\.\d+)*(\s*{)?$
* g,h: ^(\d+\.)*(sql|head)\s*=
*
* @param array $bodytextArray
* @return string
*/
private function joinLine($bodytext) {
private function joinLine($bodytext, $nestingOpen, $nestingClose) {
$data = array();
$bodytextArray = explode(PHP_EOL, $bodytext);
$nestingOpenRegexp = $nestingOpen;
if ($nestingOpen === '(' || $nestingOpen === '[') {
$nestingOpenRegexp = '\\' . $nestingOpen;
}
$full = '';
foreach ($bodytextArray as $row) {
// Valid 'new line' starts indicators: form, <level>, <level.sublevel>, <level>.<keyword>, {, <level> {, }
if ((1 === preg_match('/^\s*(\d*(\.)?)*\s*(' . TOKEN_VALID_LIST . ') *=/', $row))
|| (1 === preg_match('/^\s*(\d*(\.)?)*\s*({|})\s*/', $row))
|| (1 === preg_match('/^\s*(\d+(\.)?)+/', $row))
// if ((1 === preg_match('/^\s*(\d*(\.)?)*\s*(' . TOKEN_VALID_LIST . ') *=/', $row))
// || (1 === preg_match('/^\s*(\d*(\.)?)*\s*(' . $nestingOpen . '|' . $nestingClose . ')/', $row))
// || (1 === preg_match('/^\s*(\d+(\.)?)+/', $row))
if (($row == $nestingOpen || $row == $nestingClose)
|| (1 === preg_match('/^\d+(\.\d+)*(\s*' . $nestingOpenRegexp . ')?$/', $row))
|| (1 === preg_match('/^(\d+\.)*(' . TOKEN_VALID_LIST . ')\s*=/', $row))
) {
// if there is already something: save this
......@@ -113,14 +190,50 @@ class BodytextParser {
return implode(PHP_EOL, $data);
}
//PREG_SPLIT_DELIM_CAPTURE
/**
* Encrypt $nestingOpen and $nestingClose by a more complex token. This makes it easy to search later for '}' or '{'
*
* Valid open (complete line): {, 10 {, 10.20 {
* Valid close (complete line): }
*
* @param $bodytext
* @return mixed
*/
private function encryptNestingDelimeter($bodytext, $nestingOpen, $nestingClose) {
if ($nestingOpen === '(' || $nestingOpen === '[') {
$nestingOpen = '\\' . $nestingOpen;
$nestingClose = '\\' . $nestingClose;
}
$bodytext = preg_replace('/^((\d+)(\.\d+)*\s*)?(' . $nestingOpen . ')$/m', '$1' . NESTING_TOKEN_OPEN, $bodytext);
$bodytext = preg_replace('/^' . $nestingClose . '$/m', '$1' . NESTING_TOKEN_CLOSE, $bodytext);
return $bodytext;
}
/**
* Unnest all level.
*
* Input:
* 10 {
* sql = SELECT
* 20.sql = INSERT ..
* 30 {
* sql = DELETE
* }
* }
*
* Output:
* 10.sql = SELECT
* 10.20.sql = INSERT
* 10.20.30.sql = DELETE
*
* @param $bodytext
* @return mixed|string
* @throws UserFormException
*/
private function unNest($bodytext) {
private function unNest($bodytext, $nestingOpen, $nestingClose) {
// Replace '\{' | '\}' by internal token. All remaining '}' | '{' means: 'nested'
// $bodytext = str_replace('\{', '#&[_#', $bodytext);
......@@ -134,7 +247,7 @@ class BodytextParser {
$posMatchOpen = strrpos(substr($result, 0, $posFirstClose), NESTING_TOKEN_OPEN);
if ($posMatchOpen === false) {
$result = $this->decryptNestingDelimeter($result);
$result = $this->decryptNestingDelimeter($result, $nestingOpen, $nestingClose);
throw new \qfq\UserFormException("Missing open delimiter: $result", ERROR_MISSING_OPEN_DELIMITER);
}
......@@ -163,7 +276,9 @@ class BodytextParser {
// Split nested content in single rows
$lines = explode(PHP_EOL, $match);
foreach ($lines as $line) {
$pre .= $level . '.' . $line . PHP_EOL;
if ($line !== '') {
$pre .= $level . '.' . $line . PHP_EOL;
}
}
$result = $pre . $post;
......@@ -183,10 +298,11 @@ class BodytextParser {
* @param $bodytext
* @return mixed
*/
private function decryptNestingDelimeter($bodytext) {
private function decryptNestingDelimeter($bodytext, $nestingOpen, $nestingClose) {
$bodytext = str_replace(NESTING_TOKEN_OPEN, "$nestingOpen\n", $bodytext);
$bodytext = str_replace(NESTING_TOKEN_CLOSE, "$nestingClose\n", $bodytext);
$bodytext = str_replace(NESTING_TOKEN_OPEN, "{\n", $bodytext);
$bodytext = str_replace(NESTING_TOKEN_CLOSE, "}\n", $bodytext);
return $bodytext;
}
......
extension/qfq/tests/phpunit/BodytextParserTest.php
View file @ 05097afd
......@@ -48,8 +48,14 @@ class BodytextParserTest extends \PHPUnit_Framework_TestCase {
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one
$given = "10{\nsql = SELECT 'Hello World'}";
// Nested expression: one.
$given = "10{\nsql = SELECT 'Hello World'\n}\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one. No LF at the end
$given = "10{\nsql = SELECT 'Hello World'\n}";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
......@@ -98,6 +104,247 @@ class BodytextParserTest extends \PHPUnit_Framework_TestCase {
}
public function testNestingToken() {
$btp = new BodytextParser();
// Nested expression: one level curly
$given = "10 { \n sql = SELECT 'Hello World' \n , 'next line', \n \n 'end' \n} \n";
$expected = "10.sql = SELECT 'Hello World' , 'next line', 'end'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level angle
$given = "#<\n10 < \n sql = SELECT 'Hello World'\n>\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level angle and single curly
$given = "#<\n10 < \n head = data { \n '1','2','3' \n }\n>\n";
$expected = "10.head = data { '1','2','3' }";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level angle and single curly
$given = " # < \n 10 < \n sql = SELECT 'Hello World' \n>\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level angle and single curly
$given = " # > \n 10 < \n sql = SELECT 'Hello World' \n>\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level round bracket
$given = " # ( \n 10 ( \n sql = SELECT 'Hello World' \n)\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level round bracket
$given = " # ) \n 10 ( \n sql = SELECT 'Hello World' \n)\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level square bracket
$given = " # [ \n 10 [ \n sql = SELECT 'Hello World' \n]\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level square bracket
$given = " # ] \n 10 [ \n sql = SELECT 'Hello World' \n]\n";
$expected = "10.sql = SELECT 'Hello World'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: one level angle - garbage
$given = " # < \n 10 { \n sql = SELECT 'Hello World' \n}\n";
$expected = "10 {\nsql = SELECT 'Hello World' }";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
// Nested expression: '</script>' is allowed here - with bad implemented parsing, it would detect a nesting token end, which is not the meant.
$given = " # < \n 10 < \n sql = SELECT 'Hello World' \n head = <script> \n>\n20.tail=</script>\n\n\n30.sql=SELECT 'something'";
$expected = "10.sql = SELECT 'Hello World'\n10.head = <script>\n20.tail=</script>\n30.sql=SELECT 'something'";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
$open = '<';
$close = '>';
// muliple nesting, unnested rows inbetween
$given = <<<EOF
# $open
5.head = <h1>
3.sql = SELECT ...
10 $open
20 $open
sql = SELECT 10.20
head = <script>
tail = </script>
30 $open
head = <div>
$close
$close
30 $open
sql = SELECT 10.30
$close
sql = SELECT 10
50 $open
60 $open
tail = }
}
(:
}
]
sql = SELECT 10.50.60
head = {
{
)
{
;
[
$close
sql = SELECT 10.50
65 $open
sql = SELECT 10.50.65
$close
$close
40 $open
head = <table>
$close
$close
20.sql = SELECT 20
EOF;
$expected = "5.head = <h1>\n3.sql = SELECT ...\n10.20.sql = SELECT 10.20\n10.20.head = <script>\n10.20.tail = </script>\n10.20.30.head = <div>\n10.30.sql = SELECT 10.30\n10.sql = SELECT 10\n10.50.60.tail = } } (: } ]\n10.50.60.sql = SELECT 10.50.60\n10.50.60.head = { { ) { ; [\n10.50.sql = SELECT 10.50\n10.50.65.sql = SELECT 10.50.65\n10.40.head = <table>\n20.sql = SELECT 20";
$result = $btp->process($given);
$this->assertEquals($expected, $result);
}
public function testVariousNestingToken() {
$btp = new BodytextParser();
$tokenList = '{}[]<>()';
for ($idx = 0; $idx < 4; $idx++) {
$open = $tokenList[$idx * 2];