Sprache Part 6: DelimitedBy

This is part of a series of posts documenting Sprache:

• Sprache Part 1: Parsing Characters
• Sprache Part 2: Parsing Strings
• Sprache Part 3: Repetition (Many, AtLeastOnce, Until, Repeat, Once)
• Sprache Part 4: Or and XOr
• Sprache Part 5: Select, Return, and Regex
• Sprache Part 6: DelimitedBy
• Sprache Part 7: ChainOperator and ChainRightOperator
• Sprache Part 8: Token, Contained, Identifier, LineTerminator
• Sprache Part 9: Positioned
• Sprache Part 10: Optional and XOptional
• Sprache Part 11: Parsing Comments
• Sprache Part 12: Ref, Named, End, Not, Except, Then, Where, Preview, Concat

The DelimitedBy methods are helpers for parsing delimited sequences of elements, (like arrays [1,2,3]) with a cleaner syntax.

DelimitedBy

Parses elements matched by parser, delimited by elements matched by delimiter.

• Parser<IEnumerable<T>> DelimitedBy<T, U>(this Parser<T> parser, Parser<U> delimiter)

The following example parses a structure that looks a bit like generics type specifiers in C#:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

Parser<string> typeReference = Parse.Identifier(Parse.Letter, Parse.LetterOrDigit);

Parser<IEnumerable<string>> typeParameters =
    from open in Parse.Char('<')
    from elements in typeReference.DelimitedBy(Parse.Char(',').Token())
    from close in Parse.Char('>')
    select elements;

Assert.Equal(new[] { "string", "int" }, typeParameters.Parse("<string, int>"));

// unexpected ','; expected >
Assert.Throws<ParseException>(() => typeParameters.Parse("<string,>"));

// unexpected '>'; expected letter
Assert.Throws<ParseException>(() => typeParameters.Parse("<>"));

Note that trailing delimiters are not matched, and that the element must be matched at least once, which is where Optional comes in handy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Parser<IEnumerable<string>> array =
    from open in Parse.Char('[')
    from elements in Parse.Number.DelimitedBy(Parse.Char(',').Token()).Optional()
    from trailing in Parse.Char(',').Token().Optional()
    from close in Parse.Char(']')
    select elements.GetOrElse(new string[] { });

Assert.Equal(new[] { "1", "2", "3" }, array.Parse("[1, 2, 3]"));
Assert.Equal(new[] { "1", "2" }, array.Parse("[1, 2, ]"));
Assert.Equal(new string[] { }, array.Parse("[]"));

XDelimitedBy

Parses delimited elements, failing if any delimited element is only partially matched.

• Parser<IEnumerable<T>> XDelimitedBy<T, U>(this Parser<T> itemParser, Parser<U> delimiter)

Like all of the X variants, this is intended to improve error handling by failing early, however the difference is subtle. This example shows the difference in behaviour between DelimitedBy and XDelimitedBy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Parser<IEnumerable<string>> numbers = Parse.Number.DelimitedBy(Parse.Char(',').Token());
Parser<IEnumerable<string>> numbersX = Parse.Number.XDelimitedBy(Parse.Char(',').Token());

Assert.Equal(new[] { "1", "2" }, numbers.Parse("1, 2, "));
Assert.Throws<ParseException>(() => numbersX.Parse("1, 2, "));

Assert.Equal(new[] { "1", "2" }, numbers.Parse("1, 2a, 3"));
Assert.Equal(new[] { "1", "2" }, numbersX.Parse("1, 2a, 3"));

Assert.Equal(new[] { "1", "2" }, numbers.Parse("1, 2 "));
Assert.Throws<ParseException>(() => numbersX.Parse("1, 2 "));

What its looking for is where the combination of the delimiter + item only partially matched:

• When parsing “1, 2, " XDelimitedBy fails where DelimitedBy succeeded because it matched a delimiter, but did not match an item.
• When parsing “1, 2a, 3” however, both methods successfully matches a delimiter and an element “, 2” and stop parsing.
• Finally, when parsing “1, 2 " XDelimitedBy recognises the whitespace as a partial match on the delimiter, and so fails.