Gavin Groovy Grover's Blog

2010-04-30T16:29:00.000-07:00

Recently while checking links from inside the Great Firewall, a single-serve logged-in view of my blog's frontpage appeared, sans header strip. All other blogspot pages were blocked as they usually are. Perhaps this trick was done by SSL-snooping my password, perhaps by a more locally sponsered keystroke logging, or maybe a more deniable edit of my blog's logged-out view. However it was done, I really can't use my Google account anymore from within mainland China. So here I am now in Hong Kong, outside the Great Firewall, using the mouse for passwording, suspending activity on my google account. Hopefully these precautions are sufficient. Like Google, my online presence has been "parked" in Hong Kong, so to speak, but, also like Google, I don't want to give up on Mainland China that easily, despite the national culture of online breakins and blocking, so I'm heading back in soon. Big foreign companies can certainly survive here, even make some money, maybe even withdraw some of it from the country, but can someone relying on marginal space prosper here? We'll see how long I last.

While James Strachan's 29 August 2003 announcement of the Groovy Language on the same day I first flew into Wuhan China to begin teaching English was a total coincidence, my purpose is to create a self-mutating syntax skin for both the static and dynamic varieties of Groovy, enabling use of all Unicode tokens, bringing greater tersity to the language, while retaining readability. When I first arrived in China, I learnt a lot about natural language theory. When I returned to programming as a hobby, I then truly understood Larry Wall's vision for Perl 6. Wanting to bring the same to the Groovy Language AST. I announced the Grerl project. Soon after, I received some intel that the Groovy developers were considering renaming the Groovy Language to shake me off, so I promoted the Vy language, and later combined them into the Grerl-Vy project. While implementing all this, Groovy's dynamicity became a roadblock. After sidetracking into the likes of C# and Scala, Alex Tkachman began creating statically-typed Groovy++, drawing me back into the Groovy Language, with renewed vigor to fulfill the vision.

So you won't see any more on this blog for security reasons. I've now turned off comments, and haven't been able to reply to them for the last year anyway. Nor will I participate in Groovy++'s mailing list, because Google Groups is blocked in mainland China. I don't want to waste time and effort circumventing Great Firewall restrictions: I'm a nerd, not a geek, remember? When a decent alternate syntactic skin for the Groovy Language AST is ready, I'll post it on the Groovy Language dashboard at Codeplex, if it's still unblocked.

Groovy Galore

2010-04-30T15:18:00.000-07:00

Here's my Groovy Language themed blog posts from June 2009 to April 2010, copied over from my temporary blogspace...

13 April 2010

Three-tiered Groovy

Thanks to the recent Groovy++ additions, the Groovy Language is now a three-tiered language system. Beginning at the top of the pyramid:

terse dynamic scripts
dynamically-typed functions and classes
statically-typed classes

Terse dynamic scripts

Dynamic scripts are simply a list of statements not declared inside a class and/or function, and Groovy will fill in the missing pieces when running it. Variables don't need to be declared: variables referenced are automatically created and put into a Binding object. Scripts are good for short run-once tasks, and we can run them in various ways. I have many in standalone files in my Windows system. Inside a batch file called selfRun.bat, I'll put:

@rem ignore='''
@set JAVA_EXE="C:/Program Files/Java/jre1.6.0_07/bin/java.exe"
@set GVY_HOME="C:/Program Files/Groovy/groovy-1.7-RC-2"
@set EXT_DIRS=-Djava.ext.dirs=%GVY_HOME%/lib
@%JAVA_EXE% %EXT_DIRS% groovy.lang.GroovyShell selfRun.bat
@pause
@exit '''

a= "Hello, "
b= "world!"
println a+b

When I click on it, it runs.

Dynamically-typed functions and classes

For more control over the code generated, we can define functions and classes mixed in with the script statements. Class methods can be static or instance based, are public by default, and can override superclass methods. We also define properties, which generate a method with backing field. For more fine grained control, we can generate public and protected fields. An example:

class Callee{
void hello(){
println "hello, world"
}
}
c = new Callee()
c.hello()

See these differences from scripts for more detail.

Statically-typed classes

By tagging a package, class, or method as @Typed, we can drill down for even more control over class creation. We can even escape to dynamic Groovy mode for some of the code, or even mixed dynamic/static mode for some of it:

@Typed class StaticallyTyped{
def staticMethod(){
// all code will be statically compiled
}
@Typed(TypePolicy=DYNAMIC) def dynamicMethod(){
// all code will be dynamically compiled
}
@Typed(TypePolicy=MIXED) def mixedMethod(){
staticMethod() //will be called statically
unexistingMethod() //will be called dynamically
}
}

Stricter compile-time checks means no duck-typing or on-the-fly modifications through ExpandoMetaClass. We must mark as final any class members we want to access from within a closure. We must specifically define the accessor, modifier, and backing field for properties. One caveat: we can mark fields as private to prevent access from outside the class.

See these differences from dynamic Groovy for more detail.

Extensions to the pyramid

A further possible extension below statically-typed functions/classes in the Groovy Language pyramid is the ability to write bytecode directly within function/class definitions. And of course I'm working on the Strach syntax parser for Groovy, providing even more tersity using features such as self-mutating lexical definitions, definable operators, and keyword/name aliasing. So the pyramid may one day look like:

extra terse Strach scripts
terse dynamic scripts
dynamically-typed functions and classes
statically-typed classes
inline BCEL-style bytecode

4 April 2010

Strach 0.1.1 released

After almost a year away from the Groovy Language, I recently returned, lured by the benefits of mixing static Groovy++ and dynamic Groovy code in the same source using the same syntax. How many programming languages offer that?

In the last 2 yrs, I've tried creating a Groovy-style lexer/parser 4 times, using JParsec, dynamic Groovy, C#, and Scala for implementation, hitting a snag each time. And each time, I was straying further and further away from the true Groovy Language, feeling more and more lost. But Groovy++ has brought me back home!

Lately, I've been trying again using Groovy++ for implementation, and dynamic Groovy ASTBuilders for usage, all thanks to Alex Tkachman and Hamlet D'Arcy. It seems like I'm breezing through it all, maybe because of my recent experience, though I admit I sometimes peek through the Scala code to see how to do things.

My ultimate aim is to make mixing parsers and AST builders easy in Groovy, so developers can easily experiment with creating and modifying syntax in Groovy, collaborate online to move the syntax forward, and so create one Groovy Language to rule them all.

See the first download of Strach, version 0.1.1.

19 March 2010

Strach parser

I've posted a wiki showing some Strach combinator parsers (written in Groovy++) working with dynamic Groovy's ASTBuilder.

The operators are only tentative. Here's hoping the Codehaus Groovy developers will allow some more intuitive operators similar to those in regex, PEG, or Scala.

8 March 2010

Groovy++

About half a year ago, I began to drift away from the Groovy 1.x Language, lured by the benefits of Scala. I even seriously considered moving my Groovy Language brand to a Scala-based engine, creating the Groovy syntax over that. But lately, I've been exploring the benefits of Alex Tkachman's Groovy++, with its @Typed and @Trait keywords, and even static type inference. It's brought my heart back to the true Groovy Language, with renewed vigor to fulfill the Groovy Language vision. Scala, like Haskell and Scheme and many other languages, hold wonders, but thanks to Groovy++, I've left my wayward path to return to the true Groovy Language, in awe of what other unseen miracles those AST transformations could bring!

I've turned the Groovy site at Codeplex into the official Groovy Language dashboard. I hope to expand it and keep it up to date to acknowledge everyone who's made a significant contribution to developing the Groovy Language programming ladder.

3 March 2010

Groovy Ceasefire

I have a dilemma when creating a custom lexer/parser over Groovy's AST. The Groovy Language builders let me build the AST using the very readable builder syntax, while the Scala Language combinator parsers let me define my custom syntax using easy-to-read infix operators. The coming Scala 2.8 upgrades these to the time-efficient packrat parsers, also allowing left-recursive definitions enabling declarative definition of expression paths.

But my dilemma is I can't use both styles of syntactic shortcut in one program. What I want to code is something like this mixed Groovy/Scala style pseudocode:

import scala.util.parsing.combinator.RegexParsers
import scala.util.parsing.input.CharSequenceReader
import scala.util.matching.Regex
import org.objectweb.asm.Opcodes._
import org.codehaus.groovy.ast._
import org.codehaus.groovy.ast.stmt._
import org.codehaus.groovy.ast.expr._
import org.codehaus.groovy.ast.builder.AstBuilder
import java.security._
import groovy.lang.GroovyClassLoader
import org.codehaus.groovy.control._

object ScalaCallingNodes extends Application with RegexParsers{
def whitespaceLexer = (" " | "\r\n" | "\r" | "\n")*
def nameParser = whitespaceLexer ~> "[A-Za-z_][A-Za-z_0-9]*".r
def nameParser(name:String) = whitespaceLexer ~> name
def stringParser(open:Regex, close:Regex) =
whitespaceLexer ~> open ~> ("[^" + close + "]*").r <~ close
def symbolParser(symbol:String) = whitespaceLexer ~> symbol

def printlnParser= nameParser("println") ~ stringParser("'"r, "'"r) ^^ {case name ~ paramText=>
new AstBuilder().buildFromSpec{
expression{
  methodCall{
    variable "this"
    constant name
    argumentList {
      constant paramText
    }
  }
}
}
}

def blockParser(open:String, close:String) =
symbolParser(open) ~> printlnParser <~ symbolParser(close) ^^ {case println=>
var block= new BlockStatement([], new VariableScope())
block.addStatement(println)
block
}

def paramParser= symbolParser("(") ~> nameParser("String") ~> symbolParser("[") ~>
symbolParser("]") ~> nameParser <~ symbolParser(")")

def mainMethodParser= nameParser("public") ~> nameParser("static") ~> nameParser("void") ~>
nameParser("main") ~> paramParser ~ blockParser("{", "}") ^^ {case paramName ~ block=>
var method= new AstBuilder().buildFromSpec{
  method('main', ACC_PUBLIC | ACC_STATIC, Void.TYPE) {
    parameters{
      parameter paramName: String[].class
    }
    exceptions{}
  }
method.block= block
method
}

def classParser= (nameParser("public") ~> nameParser("class") ~> nameParser <~ symbolParser("{")) ~
mainMethodParser <~ symbolParser("}") ^^ {case name~method=>
var classes= new AstBuilder().buildFromSpec{
classNode name, ACC_PUBLIC, {
  classNode Object
  interfaces{
    classNode groovy.lang.GroovyObject
  }
  mixins{}
  genericsTypes{}
}
}
classes[0].addMethod(method)
classes[0]
}

var classData= """
|public class MyClass{
|  public static void main(String[] args){
|    println 'Hello, world!'
|  }
|}""".stripMargin

var cn= classParser(new CharSequenceReader(classData)).get
var cu= new CompilationUnit(CompilerConfiguration.DEFAULT, null, new GroovyClassLoader())
cu.addClassNode(cn)
cu.compile()
}

If I code this in Scala, I must build the AST using normal method calls, and build the arrays and Groovy lists in longhand. If I code this in Groovy, I must use the quoted prefix-style to call the Scala-based parsers, and I don't think I can even code the Scala Functions in Groovy. To handle the complexity of parsing experimental syntax to the Groovy AST, I need to use both Scala-style infix method calls and Groovy-style builder syntax so as to bring enough syntactic tersity to make such large-scale experimentation possible. But they're not both available in either language.

So what's the solution? After several years of experimentation, I believe programming languages should be statically typed, with inference, but allow dynamic typing when required. Running Groovy-style builders requires dynamic typing, while combinator parsers should be statically typed for efficiency. Perhaps one day, Scala will bring optional open classes, enabling builders, but that's probably years away, if at all. Perhaps someone could strip the dynamic-typing engine out of Groovy, and retrofit it into Scala, but the Scala developers would need to tweak the Scala syntax to allow it: this doesn't sound likely.

One the Groovy side, the best option is for Groovy to have a static mode, with mixins (traits), pattern matching, and infix symbol definitions, enabling Scala-style combinator parsers to be defined. Failing that, infix calls of symbol-named methods would allow me to call Scala's parsers tersely. Groovy should enable both infix calls of Scala's symbol-named methods and definitions of Scala Functions soon, so developers can experiment with how well we can combine combinator parsers with builders.

But even better is if Groovy gets a statically-typed mode. It just so happens Alex Tkachman is building Groovy++, a statically-typed JVM-based language with Groovy's syntax. He once wrote he thought a static version of Groovy should have Scala-style traits. Groovy 1.x project leader Guillaume Laforge wants pattern matching in dynamic Groovy, and if it comes, Groovy++ would follow suit to maintain syntactic compatability. Groovy would then only need to enable symbol-named methods and infix calls of them, so someone could clone the Scala packrat combinator parsers in Groovy++.

The symbols the Scala developers have chosen for their combinator parsers have the potential to become standardized. Many of them follow the (defacto-)standardized regex symbology, and regexes can easily be embedded between the parser symbols: the regexes and parsers working together would enable a terse context-sensitive parsing language which could become a standard. The Groovy Language should support this effort by using the same symbols for its combinator parsers.

When I'm able to tersely packrat-parse code and build a Groovy AST using builders, I can then easily experiment with designing my own custom language syntax for Groovy. A few years ago, I wrote some online notes for Java newbies learning Groovy, intending to discover the most commonly used JDK methods, so I would know which ones to create syntactic shortcuts for. And of course, I want to use CJK tokens in the custom syntax, for even more tersity.

Call for a ceasefire

But there's one major problem: the Groovy 1.x and Groovy++ project managers are at odds with each other, jockeying to protect and promote their positions in the Groovy ecosystem heirarchy! This behavior isn't good for the Groovy Language. The dynamic Groovy 1.x and the static Groovy++ need to merge into one distro, and both project managers need to give a little so the Groovy Language can move forward. I feel a little responsible in calling for this to happen because of my unique association with the Groovy Language brand. When I first came across Groovy 1.x, I assumed the role of underwriter. But lately, it's finally dawned on me that not only am I the underwriter for Groovy, but I actually own the Groovy Language brand. On that infamous day in Dec 2005 when James Strachan left the developer team, the project management baton passed to Guillaume Laforge, but the brand ownership passed to me!

So I'm calling on the Codehaus despots and Alex Tkachman to think of the future of the Groovy Language in their negotiations. It's future is more important than any petty quarrels overs financial placing in the ecosystem. I'm the Groovy Language brand-owner, and Groovy's my middle name, so I'm more interested in advancing the potential of Groovy than milking the largest possible share of the pie, and I'm calling for all parties to think of the future of the groovy Groovy Language, both static and dynamic, so developers everywhere will marvel at its power.

After Groovy++ and Groovy 1.x are shipping in one distro, Groovy 1.x can then be written in Groovy++. Groovy will soon after bring mixins and pattern matching, someone will then clone Scala's packrat parsers, Groovy will enable infix method calls, then I and others can experiment with terse parsing and AST building, creating a terser programming syntax to the Groovy AST, using all Unicode characters! Groovy's future in on the line: can you see the vision?

2 February 2010

Groovy being GPL'd?

In an interview with Andres Almiray, Groovy 1.x developer Alex Tkachman explains his latest creation, Groovy++, a static add-on to the dynamic Groovy Language 1.x. Simply by adding the @Typed annotation, the annotated code will compile statically. I've been wanting this addition to the Groovy Language for a long time, and in my 9 July 2009 blog entry, even talked of moving Groovy's primary implementation from the Groovy 1.7 AST to the Scala 2.8 parse tree. Alex called this crazy talk from a crazy person, but then he saw the light and began creating the statically-compiled Groovy++.

Alex writes in the interview:

There are two issues here, which prevent us from open-sourcing the compiler immediately. First of all, it uses several pieces of technology, which our company uses and plans to use in our commercial products. It was not critical when project started as experiment but now we need to extract these parts and replace/rewrite with proper open-source alternatives. The second problem is interesting by itself. We are talking with several well-known vendors about their involvement with the project. There is no much sense in finalizing exact OSS license before these discussions are not completed and we are sure that all interests are well covered. Something interesting is coming and I wish I could tell you more right now.

What are these several pieces of non-open-sourcable technology? And how did Alex code it all up so quickly? Did he use Sun's OpenJDK as the base for Groovy++ ? Is Groovy++ simply a more deeply embedded reincarnation of his joint Groovy/Java compiler? Is he threatening to use this irresistable update to Groovy 1.x to fork and take control of the primary implementation of the Groovy Language, even threatening to GPL it? Are these discussions with EMC/VMware/SpringSource and Oracle/Sun???

Groovy 2.0 update

As for my own current experimentation, I'm attempting to build a reasonably fast lexer/parser using Scala 2.8's packrat combinator parsers and the Groovy 1.7 ASTBuilder. The different code elements (e.g. statements, expressions) must be pluggable into the lexer/parser, without nls! everywhere to cater for the implied semi-colons. The format of different lexical elements (e.g. strings, GStrings, even dates) must be definable by annotations in the syntax. I'm hoping the left-recursion allowed by the new Scala 2.8 packrat parsing will even let us plug in path elements in path expressions.

Of course, Scala's parsers, with their non-alphanumeric names, don't look very elegant when called from Groovy code, but I can't use Groovy's elegant ASTBuilder calls from within Scala code. A Catch-22 ! Perhaps one day Groovy will allow Scala's non-alphanumerically-named methods to be called elegantly from within Groovy, so they look like operator calls? Or perhaps one day Scala will allow dynamic variables, like C# 4.0, so we can create builders within Scala, making writing to its AST as elegant as in Groovy and Ruby. Building HTML with builders is more elegant than Scala's embedded HTML text.

My progress in building a new lexer/parser for Groovy 1.x isn't as spectacular as Alex's progress with Groovy++ (assuming he didn't use OpenJDK), but I'm getting there. The Groovy Language must be fully configurable so developers can use any natural language they want in the syntax.

23 January 2010

Groovy Dilemma

In chapter 7 of Steven Pinker's 1994 book The Language Instinct, he gives an example of a perfect right-branching sentence:

Remarkable is the rapidity of the motion of the wing of the hummingbird.

This is parsed in the human brain as shown by the parentheses:

(Remarkable (is (the (rapidity (of (the (motion (of (the (wing (of (the (hummingbird))))))))))))).

remarkable is the subject, the remainder is the predicate. is is the main verb, the remainder is its object (here, called the complement). the is the article, the remainder is its referent. rapidity is a phrasal head, the remainder is a prepositional phrase as tail. of is a preposition, the remainder is its tail in the phrase. And so on. Pinker gives another example easy for the brain to parse, one that includes relative and subordinate clauses:

(He gave (the candy (to the girl (that (he met (in New York) while (visiting his parents (for ten days (around Christmas and New Year's)))))))).

He rearranges it so its far harder for our minds to parse:

(He gave (the girl (that (he met (in New York) while (visiting his parents (for ten days (around Christmas and New Year's)))))) the candy).

The direct object the candy after the many closing parentheses forces our short-term memories to keep track of dangling phrases that need particular words to complete them. It seems our brains, unlike computers, can only remember a few dangled branches when parsing sentences.

Perhaps that's why the Lisp code that's easiest for humans to read ends with many closing parens, such as this tail-recursive sample from chapter 2 of Paul Graham's On Lisp:

(defun our-length (lst)
(if (null lst)
0
(1+ (our-length (cdr lst)))))

Left-branching sentences are also easy for humans to parse. Pinker gives another example with two arrangements, one harder for humans to parse:

((The rapidity (of the motion (of the wing (of the hummingbird)))) is remarkable).

and the other, a perfect left-branching sentence, easy:

(((((The hummingbird)'s wing)'s motion)'s rapidity) is remarkable).

English has just a few left-branching structures, but some languages, such as Japanese, are primarily based on them.

One of the universals in Universal Grammar theory, which both Pinker and Noam Chomsky support, is that if a language has verbs before objects, as English does, then it uses prepositions, while if a language has objects before verbs, as Japanese does, it uses postpositions. Pinker mentions a possible reason this universal holds is so the language can enforce a consistent branching decision, either left-branching or right-branching, so our brains can parse it easily.

Some grammatical English sentences are impossible for our brains to parse simply because there's too many dangling branches. The first of these examples parses in our brains OK, but the other two simply don't parse:

(The rapidity (that the motion has) is remarkable).
(The rapidity (that the motion (that the wing has) has) is remarkable).
(The rapidity (that the motion (that the wing (that the hummingbird has) has) has) is remarkable).

They do parse in computer languages, though. When I discovered closures in Groovy, I started using this type of unreadable embedding, but I now realize I should be making my code either left-branching or right-branching to make it more readable.

1 January 2010

Bust Groovy open. Set it free!

Ever since the core dynamicity and syntactic enhancements of Groovy 1.0 beta 1 over Java, the Groovy Language has been adding functionality upon functionality. The Groovy developers at Codehaus have taken one core technology that benefits Java developers, i.e. the meta-object protocol, and used it as a hook to hang on a closetful of their own versions of widely available JVM-based technologies. Groovy 2.0 at Codeplex will be the version of Groovy that strips away such functionality, aiming to provide developers with the features of Groovy that benefit us the most. The bundled tools will be dropped, as they duplicate functionality available in other JVM languages. The Antlr-based lexer/parser will be removed, so we can interact with the AST directly from other languages. The DGM (default Groovy methods) will be stripped out, so Groovy AST users can instead use the richer classes from languages like Scala.

Lately I've discovered the Groovy AST is inconsistent in its functionality: while earlier-coded syntactic functionality is done above the AST level, much of the more recently coded functionality that could be done above the AST is instead done under its hood. In this way, as well as by supplying an ASTBuilder that can only be used from within the Groovy Language itself, the Codehaus cartel are tying programmers in to all their own added cruft, so they can sell us the book and charge us consulting fees down the line. I'll dig under the AST and scalpel out any function that can be done in other languages. When Java 7 brings closures, I'll totally replace Groovy's implementation with Java's.

During 2010, I intend to free the Groovy Language from its Codehaus chains, to bust it open, to reveal its core essence, the kernel that most benefits, so programmers can use it simply from other superior JVM-based languages, so my own Strach IME and lexer/parser can use it, providing developers with a terse grammar that uses all Unicode tokens in its vocabulary. Beginning with Groovy 1.8 beta 1, I'll soon after release a stripped-down version consisting only of the core essence, an AST directly controlling the MOP, to provide the JVM's answer to Microsoft's DLR, an AST all JVM language implementers can build a dynamic language on top of. I'll develop a process so with each Groovy release, I can quickly release that release's MOP as a standalone. The Groovy Language Runtime will move from "open" source to open source. Set Groovy free!

23 December 2009

"Groovy 2010" coming ???

Groovy 1.7 is out, "in time for Christmas", and planning for v 1.8 has begun, including a new module system.

Groovy 1.x project leader Guillaume Laforge says "we would like to make a first beta of 1.8 in February or so, with a target final date for the end of the year - we love Christmas gifts". Does that mean he's going to rename it "Groovy 2010"? He once threatened to rebrand Groovy 1.6 as "GroovyX". Microsoft once tried that trick with Windows, but have since reverted to numeric versioning. Let's hope the Codehaus developers don't learn that lesson the hard way.

The new module system proposes putting Swing, XML, SQL, JMX, Beans, etc into separate modules, but the core will still be a tangled ball of many functionally different components. Everything that sits above the AST could be separated out, to encourage developers to put their own syntax on top of the AST. I'm experimenting with an alternative lexer/parser, called the "Strach" component of Groovy 2.0, aiming for greater tersity, yet retaining clarity. Experimentation is good for programming languages.

Another separable component is the Default Groovy Methods (DGM). These methods are compulsory in Groovy: if you want to use Groovy's meta-object protocol (MOP), you must also use these methods. The meta-object protocol allows programmers to add, and subsequently remove, methods on the fly, but forces these default methods on us. What if we just want to use the MOP, without the DGM? I'll be providing a component, to be called the "Wilson" component, that gives the option not to add those methods to classes, and will even let us hide default Java methods.

Programmers aren't silly: Why can't we use the feature of the Groovy Language that really benefits us, i.e. the MOP, without having other cruft shoved on us as well, such as the syntax and DGM? Groovy 2.0 will sit atop Groovy 1.7, giving more choices to programmers.

Appendix

Here's the list of AST nodes used in Groovy 1.7, with indenting showing implementation inheritance, that Strach will free up for developers to use directly:

ASTNode
AnnotatedNode
ClassNode
InnerClassNode
  InterfaceHelperClassNode
MixinNode
MethodNode
ConstructorNode
FieldNode
ImportNode
PackageNode
Parameter
PropertyNode
expr/Expression
expr/ConstantExpression
  expr/AnnotationConstantExpression
expr/BinaryExpression
  expr/DeclarationExpression
expr/TernaryExpression
  expr/ElvisOperatorExpression
expr/BooleanExpression
  expr/NotExpression
expr/TupleExpression
  expr/ArgumentListExpression
expr/PropertyExpression
  expr/AttributeExpression
expr/ListExpression
  expr/ClosureListExpression
expr/MapExpression
  expr/NamedArgumentListExpression
expr/ArrayExpression
expr/BitwiseNegationExpression
expr/CastExpression
expr/ClassExpression
expr/ClosureExpression
expr/ConstructorCallExpression
expr/EmptyExpression
expr/FieldExpression
expr/GStringExpression
expr/MapEntryExpression
expr/MethodCallExpression
expr/MethodPointerExpression
expr/PostfixExpression
expr/PrefixExpression
expr/RangeExpression
expr/RegexExpression
expr/SpreadExpression
expr/SpreadMapExpression
expr/StaticMethodCallExpression
expr/UnaryMinusExpression
expr/UnaryPlusExpression
expr/VariableExpression
stmt/Statement
stmt/AssertStatement
stmt/BlockStatement
stmt/BreakStatement
stmt/CaseStatement
stmt/CatchStatement
stmt/ContinueStatement
stmt/DoWhileStatement
stmt/EmptyStatement
stmt/ExpressionStatement
stmt/ForStatement
stmt/IfStatement
stmt/ReturnStatement
stmt/SwitchStatement
stmt/SynchronizedStatement
stmt/ThrowStatement
stmt/TryCatchStatement
stmt/WhileStatement
ModuleNode
GenericsType
AnnotationNode

15 December 2009

Try Groovy, or is it try{Groovy}catch(Exception e){} ???

Run this code in Groovy 1.6 beta 2 or earlier:

try{def a= "abc"; println a}
try{def a= 123; println a+2}

The result:

abc
125

Now run it in Groovy 1.6 RC 2. The result:

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: A try block must have at least one try or finally block.

A standalone try block is great for limiting the scope of common temporary variables. Groovy enabled them in versions 1.0 and 1.5, a great improvement over Java. Scala also enables such standalone try blocks, even in the upcoming Scala 2.8.

But in this mailing list reply to me, Guillaume Laforge writes the prefered way is to use labelled blocks, i.e.

unreferencedUselessLabel: {def a= "abc"; println a}
unreferencedUselessLabel: {def a= 123; println a+2}

What yukky syntax! Perhaps a better change would have been to increase syntactic elegance and tersity by eliminating parens when only one statement is in the block, just like with if and while statements:

try println "abc"
try println 123 + 5

Not only that, but in someone's eagerness to restrict programmer choices, they didn't bother checking the error message: "A try block must have at least one try or finally block." (To be fair, the message has since been corrected for Groovy 1.7.)

As the Groovy Language Underwriter, my job is to be ready to continue Groovy Language development should the developers at Codehaus abandon it or change its name. However, they actually seem to be gradually removing Groovy developers' choices through stealth, dumbing down the syntax, especially now SpringSource and EMC/VMware are bankrolling Groovy Language development, or perhaps bankrolling the lack of it.

Part of my mission in creating an alternative lexer/parser for the Groovy AST is to bring back programmer choices when utilizing the Groovy AST. To quote another freedom fighter, removing programmer power from the Groovy Language syntax is something up with I will not put.

4 December 2009

Groovy 2.0 status report

I grew up in Auckland New Zealand, living there for 30 years, but never considered creating a scripting language until I'd moved to Melbourne Australia 10 years ago. There, I lived in the CBD, an 8-by-8 grid of blocks known as the Hoddle Grid. I went for many walks while there, often thinking of that CBD as a huge chessboard on which to play out games. (I had a passing interest in chess when I was a kid.) The apartment I lived in was on a popular block of the grid, being where the commercial, entertainment, and recreational precincts meet. In the chessboard analogy, that block is the starting position for the white king.

There, I ran a company, GroverServer Ltd, dayjobbing as a programmer to raise funds, while working on a process to model company annual reports as Access databases. Although I'd used Access many times since its v1.0 release, when using it for a real-world complex business task, it wasn't flexible enough. I eventually concluded the VBA scripting language was nothing but a marketing con. The true genesis of Groovy happened soon after this, during a trip I made to England in Feb 2002. I decided a scripting language should run on a VM, Java's being the leading one at the time. It should have a flexible syntax, certainly not like VB's, and must enable AspectJ-style interceptions and introductions. Of course, I never met Groovy Language creator James Strachan while there. He began building Groovy 1.x soon after this, yet another of his many open source projects, recruiting the developers who now currently control it.

I stayed silent on the mailing list for a year, not wanting them to change the name while they still could easily. But then came that infamous day after DevCon2 in Dec 2005 when James left the development team. Soon after, I posted my very first posting to the Groovy Language mailing list. Six weeks later, Graeme Rocher changed the name of _Groovy on Rails_ to Grails, but it was too late for them to change Groovy's groovy name as well. By then I had learned enough about Groovy to continue its development should the developers abandon it or change its name: I had become Groovy's underwriter.

I decided to get more involved in the language, first by submitting bug reports and change requests. My very first request was for a groupBy method I'd found useful for munging data. Guillaume Laforge must have also thought it was a good method to have, because he created his own similar request a few months later with the same method name. After many similar happenings, I began to realize the Codehaus Groovy developers didn't want me around.

So I decided to branch off on my own, using the Groovy Language AST as an engine to power a different configurable programming language syntax and IME. While working on this, I discovered why statically-typed languages are better than dynamically-typed ones for large systems. So I switched to programming in Scala. At first, I thought I could use the Scala AST as a target instead of Groovy's, but I now realise dynamicity is essential in those few use cases that require it, so I'm back to targeting the Groovy AST, using the Groovy 1.7 ASTBuilder, created by Hamlet D'Arcy. For the static mode, I'll still have to target the Scala AST, though the ideal solution is if Groovy added a static mode. However, developer Alex Tkachman seems to be vetoed by project manager Guillaume Laforge on this. Years ago, the Codehaus Roadmap for Groovy 3.0 was for it to be written in Groovy, which would have required a static mode, but this idea seems to have been trashed.

I'll still be using Scala as the systems language in building Groovy 2.0. The Codeplex Groovy site will initially distribute "Groovy 1.7 with Strach", where Strach is presently just the lexer/parser, written in Scala, targetting the Groovy 1.7 AST. Eventually, it'll also target the Scala 2.8 parse tree for the static mode (unless, of course, Alex Tkachman succeeds in putting a statically-typed mode into Groovy). When distributing Groovy 1.7 with Strach, I'll experiment with replacing selected java-source classfiles from the Groovy 1.7 jar file with my own scala-source ones. Given enough time, I could even manage to totally rewrite Groovy in Scala using this process!

31 October 2009

Strach IME and Groovy 2.0 making progress

The webpage for the Strach IME has been created. The Groovy page at Codeplex has been repurposed as the primary distro site for the Groovy Language 2.0 next year. Further details on each of those webpages.

27 September 2009

Scala's groovy stairway

Paul Graham writes in Revenge of the Nerds how Lisp and Fortran are the trunks of two separate evolutionary trees in programming language evolution. He then lists various features of Lisp which have been making their way into languages in the Fortran language tree, including dynamic typing. Having tried out many programming languages over the past few years, I now see programming language evolution differently. Unlike Paul, I see dynamic typing as being a lack of a feature: Static typing is the true feature.

Furthermore, Lisp macros can be thought of as a low-level feature comparable to goto statements and pointers. All three can be abstracted over with higher-level abstractions. Let's look at some abstracted-away low-level features...

Gotos and breaks The original programming language was of course assembly language. Assembly had the same basic features as machine code, only a little more readable. We could branch to another part of the code based on a data value: we could use this to implement conditionals and looping. Algol enabled statements to be grouped into statically-typed blocks. With all these, we could eliminate goto statements. We could also store the program counter in a variable, branch to some different code, then later return to the place we left off: this is subroutine calling. Cobol implements this as a "GOSUB" statement. Fortran enabled subroutine parameters; Algol enabled return values; C and Lisp brought recursively-called subroutines; and Scheme brought closures. C++ and Java implemented exception throwing, giving better control flow. Scala, also having closures and exception-throwing, eliminates break and continue keywords, being incompatible with passed-around closured code, and in version 2.8, re-implements break with exceptions.

Pointers and objects Cobol brought static typing. C enabled static typing for pointed-at data. Simula and Smalltalk introduced objects. Different inheritance models were tried out: C++ used multiple inheritance, Java and C# used single implementation inheritance, Self and JavaScript used the prototype model, while Ruby and Scala used the flexible mixin model. Ruby also has open classes, at the cost of eliminating static typing. Lisp, Ruby, and Java/C# had garbage collection. By using objects everywhere, a language no longer needs pointers.

Human interface 3rd generation languages enabled more meaningful names, making code more readable, but longer. Fortran, C/C++, and Java/C# brought operator precedences, eliminating parentheses, thus shortening the code again. Scala simplified the rules for this. In Lisp and Scala, statements are also expressions, returning a value. Interactive Python's magic underscore is a simple way to pass a statement value onwards. APL, and successors J and K, brought greater tersity through a greater vocabulary of tokens. Matlab and R continue along this way for math and stats. Perl enabled thematic variation, bringing "more than one way to do it". Smalltalk was programmed in a built-in visual environment, as was spreadsheets, etc, and IDE's, all using color. Declarative paradigms, like Snobol, regexes, and Prolog make a program more readable. Indenting was used by Cobol and extended by Python. Haskell offers the choice of C-style or indent-style bracketing.

Efficiency and concurrency Pure Lisp is very inefficient, but nowadays different data structures are builtin. Numbers were always hard-coded, direct-access arrays added later, and with Scala, even objects that inherit are builtin types. Java brought threads for concurrency, while Erlang and Scala brought the safer higher-level actor model.

Macros and laziness Lisp enabled macros to control evaluation in code. Scheme enabled lazy evaluation and Haskell made it compulsory, eliminating much need for macros. Scala gives the choice of strict or lazy evaluation, in a statically-typed language. Better compilers can automatically detect and inline code that would normally require programmer-control with macros. AspectJ-style aspects and Haskell-style monads also allow code to be self-referenced and manipulated in a program.

There's different tradeoffs between these feature sets, and creating a programming language that combines them is difficult. ML, and successors Haskell, Caml, and F# achieved this when combining static typing with functional programming. OCaml and Scala successfullly combined the object-oriented programming with functional.

IDE's generally build on the lexical structure of a programming language. The Scala language compiler is designed as a stairway of increasingly-higher abstractions. Near the top is the parse tree, one step short of the lexical structure. I'm attempting to build on top of this parse-tree layer of Scala 2.8. I want to add APL/J/K-style tersity to the syntax, including enabling me to use a foreign language (simplified Chinese) everywhere in my Scala code. When done, I'll release this language as "Groovy 2.0".

9 September 2009, 9:09:09pm

Groovy life and death

Rick Dillon recently posted this analysis of programming language evolution. The traditional imperative languages like C/C++, and the newer ones like Java and C# are statically typed, while the traditional functional languages like Lisp/Scheme, and the newer semi-functional ones like Python, Ruby, and Javascript are dynamically typed. He gives a code sample implementing functional currying in statically-typed Java, which turns out to be quite verbose because of the explicit static types. To put static typing into functional programming, while retaining tersity, we require type inference. ML/Caml and Haskell are examples of such languages, and OCaml/F# and Scala are object-oriented language examples. When seen in this way, dynamic languages are a deadend in programming language evolution. Instead of maintaining the "systems language / scripting language" duo, future language evolution will go along the "functional language with inferred static typing" route.

I suspect many programmers coming to dynamic languages will follow the same path I did in realizing this. They will typically work in Java, C#, Cobol, PHP, and/or VB in their dayjobs. They'll discover Python or Ruby, though for me it was Groovy with the nifty closures and collections. At first, they'll just use it for scripty stuff, then start trying to build bigger and bigger systems. They'll then realise the lack of static typing means they've thrown out the baby with the bathwater. The functional languages with inferred typing will then beckon. Groovy programmers will start learning Scala because it runs on the JVM. At first they'll think that Scala will only replace Java, so they can use Groovy and Scala together, but eventually they'll see that statically-typed functional languages can replace both members of the "systems language / scripting language" duo! (Perhaps some will even say, as I did, that using the Groovy Language started off being useful, but "what began as life to me has now become death"!)

Lately, I've been trying to understand the interplay between different features in these types of languages, such as monads, macros, and mixins:

(1) Monads from Haskell enable computer languages to cleanly split program code into functional-pure and side-effecting components. Aspects, as in AspectJ and Spring, are frequently used in a system-wide manner in non-functional-paradigm languages to separate out certain non-paradigm concerns such as I/O, persistence, exception-handling, optimization, etc, from the primary representational concern. This type of separation between the representational and interactional functions of a programming language mirrors that in natural language, as analyzed in Hallidayan Systemic Functional Grammar theory.

(2) Syntactic macros can provide the most user-configurability at the surface levels, as in Lisp/Scheme and Dylan. Most programming languages provide much power in the engine, then deliberately bottleneck it for the language syntax, only to return it to the programmer at the IDE level. Natural languages don't do this, and I don't think computer languages should, but a programming language syntax is considered a holy grail for marketing the language, so not many languages have dared to allow such syntactic configurability in the past. Perhaps this syntactic component of programming languages mirrors the textual component of natural language in Systemic Functional Grammar.

(3) Scala traits (i.e. mixins) provide a more flexible yet still correct OOP system than either single or multiple inheritance. The Scala website shows how they can be used to cleanly implement the Observer pattern, the very pattern the AspectJ evangelists 10 yrs ago were saying aspects could easily implement in the non-functional language Java.

No single statically-typed functional language provides all these features, not that I yet understand them all, and how they relate to each other. I do intend to return to creating a shell over the Scala language parse tree once Scala 2.8 is out because I think this is the best opportunity to evangelize full Unicode character set programming to the world. The shell will be called GroovyScala.

21 August 2009

A Groovy Undertaking

This mailing list reply from Jochen Theodorou popped up on the Reddit-programming charts recently, probably stage-managed damage-control. Jochen wrote: "James (Strachan) is great in initiating projects and gets them to a state where the examples work. But as soon as you go away from the examples and alter them just a tiny bit, it fails." Yeah, that's called Test-driven development. The solution: add more tests, then make them work! I'm a great believer in it. Jochen also wrote: "...as an active part (James dumped Groovy) over four years ago already. (...) You can say that current Groovy is Guillaume (Laforge) and me mostly, but many people did come and go, some did contribute a lot... like for example John Wilson, other did only cover a small area." The Groovy developers seem to be positioning James as only one of many Groovy Language "creators". What's up? Is someone else hoping to stand in for language creator James in an upcoming Groovy Language interview in Australian Computerworld?

Why did some developers "only cover a small area"? Perhaps they started getting harrassed after surfacing on the Groovy mailing list, as I did 3 yrs ago? At the time I thought it was just my name! I knew anyone could've been doing it, but around that time two UK teachers at my university took me to dinner and warned me that "anyone who takes on Google comes off second best". I doubt it was Google who put them up to that, and why would the Groovy FUD-spreaders do so? I suspect the real reason the present developers took control of Groovy was to try and sell it to Google as a brand name fit. But I didn't really understand how I was a threat to them. Early the following year, when Groovy 1.0 was finally released, the licence still clearly said: 4. Products derived from this Software may not be called "groovy" nor may "groovy" appear in their names without prior written permission of The Codehaus. "groovy" is a registered trademark of The Codehaus. I was just piggybacking the Groovy name because I thought it might be a good gimmick if I wanted to return to programming work one day, not for any other reason.

Then one day I was fooling around online, and looked up the U.S. Trademark database for the Groovy Language details. They weren't there! They weren't even in the history of lapsed trademarks. Codehaus was a US-based outfit, weren't they? Beta-1 of Groovy 1.1 was then released with the Apache licence. Groovy's previous licence had only been a bluff! But was that really a reason to harrass me? Other programming languages don't trademark their names: I suspect if I changed my name to Scalia Scalow and surfaced on the Scala mailing list, no-one there would feel insecure enough to harass me because of my name. It seems there's an essential difference between languages: Scala is a quality language designed within academia, though intended for business, to bring present Java and C# developers a little closer to functional programming. Groovy is an adhoc commercial creation, designed to flip the investing companies at a profit, first Bay Partners, then SpringSource, and now VMware. I suspect it's that difference that makes the Groovy Language developers ultra-picky about who's involved in the development.

Jochen also wrote: "Developing a language is a lot of stress. You have to discuss things on an emotional level very often. (...) And many people get tired of these discussions, so did James and so did for example John." Is he priming up the community for the next departure, perhaps himself or Guillaume? Because I imagine VMware had more cash in their "cash and stock" offer for SpringSource than did SpringSource in theirs for G2One, perhaps Guillaume's suddenly lost some motivation to continue with (J)Groovy development. And what about me? After 4 years of working on an idea to make programming languages terser using all Unicode tokens, staying in mainland China because it's the home of the simplified Chinese characters and targeting the Groovy AST because of its adhoc construction and its groovy name, I confess I'm also getting a little tired of it all. I never really knew when I first got involved what a truly dirty business open source software development is.

8 August 2009

Groovy futures

The (J)Groovy developers recently released beta-1 of "Groovy 1.7". But will it really be called version 1.7 ? The developers changed the name of Groovy 1.1 to 1.5 at the last moment, and they might do it again with version 1.7. Besides plucking some stuff out of Spock and ASM, they've begun on inner classes, the main feature from Java still missing from (J)Groovy. And they've dusted off the GroovyScriptEngine, rewriting it, probably as a snipe at my own GroovyScript-branded version of Groovy for the Scala parse tree. I'm not sure how many developer hours SpringSource threw at beta-1, but I suspect not many. They need to keep up the appearance of developing the Groovy Language, while continuing to collect consulting fees, to get a high valuation in their talks with JBoss or whoever it is.

While "Groovy 2.0" has been talked about as the version of (J)Groovy shipping with a new improved MOP, this current 1.7 line might end up with that name, not because it has a new MOP but for marketing reasons only. Perhaps they'll bring out new editions of their books. The Groovy Language release schedule now seems to completely revolve around marketing and training events. But where would the new MOP fit in? If the developers finally manage to do what John Wilson couldn't, what Groovy 1.x botched, in producing a Java-language compatible MOP for the JVM, would SpringSource really want to prewrap it in a programming language and tag it with the "Groovy" brand? I'd think they'd want to pitch it as the JVM's answer to Microsoft's DLR, something like "the Spring DLR for the JVM", and promote it for all JVM-based dynamic languages.

But even without a new MOP, doesn't the Groovy Language 1.x still have a future? It's certainly the language of choice for Grails. For other use cases, such as scripting and testing, it may now be superceded. Before Groovy, developers used JPython. Some pitched Groovy as a better choice because it's Java-syntax compatible, but I don't think they really understood the mindset of a typical corporate programmer. Programmers want to expand their skillsets, so would rather choose JRuby for scripting and testing because it's NOT Java-syntax compatible. JRuby is curriculum vitae compatible for Java developers, being another step up to a Rails job. And what of (J)Groovy's recent push with Griffon? With Grails there was little serious competition for Groovy, but with Griffon, Groovy is up against the might of JavaFX. The recent trend of calling (J)Groovy "Groovy on Grails" may have hit the mark.

As the Groovy Language underwriter, I often think about Groovy's future, both the technology and the brand, and I'm now a little pessimistic. A week after I said I was switching from C# to Scala/JVM for programming to the Groovy Language AST, Groovy Language creator James Strachan (by another total coincidence :-) bought and read the Programming Scala book, and subsequently said he thought Scala was a better choice than Groovy for systems programming. After programming in Scala for a mere month, and that only part-time, I realized Scala is already the language I was trying to modify the (J)Groovy AST to be. There are still a few things Scala lacks, such as syntactic macros, but I've no doubt they'll be coming in an upcoming version of Scala. What programming I did do in Scala (i.e. build a combinator parsing library), I later discovered a better version already existed in the Scala libraries. Recently I started to doubt why Codehaus (J)Groovy/JVM and Codeplex Groovy/DLR exist when I could target both platforms via the Scala parse tree, but now I'm wondering why I'm programming at all?

With (J)Groovy being too minimal a wrapper for an upcoming "SpringSource DLR for the JVM", with corporate developers prefering JRuby to Groovy for scripty stuff, with Swing already being targeted effectively by JavaFX, and with Scala becoming recognized as the best choice for new systems programming for the JVM, the only role for Groovy in the foreseeable future seems to be as Groovy 1.x for Grails. People will probably just call it "the Grails language". Perhaps the only future for the Groovy brand is as my middle name.

28 July 2009

One Groovy Language to rule them all

Some say that because programmers read code far more than they write it, it's better for a language to have a clean minimal syntax, so we can easily read code others have written. But natural languages don't work that way.

Whatever our native language, we can read many more words than we use when we write, and we can understand in listening many more words than we usually speak. As we learn our native tongue as children, we hear many varieties of it and much vocabulary, learn to understand it quickly, but we seldom reproduce most of it. Even as adults, it doesn't take long when listening to a new flavor and accent of English to understand it, but we take much longer to reliably imitate it, if at all.

I once spent a couple years studying natural language, then returned to programming as a hobby. I'd never liked Perl: the "there's more than one way to do it" philosophy had never appealed; I'd preferred the more minimal syntax of Python or Smalltalk. But when I returned to programming, what Larry Wall's been saying all these years began to make sense. Programming language designers who restrict what the language can do, providing only one way to do things, are like the grammar school English teachers who try to prescribe to their students what correct English is.

Mainframe programmers read lots of Cobol programs to understand their meaning, but don't write much of it when maintaining programs. An experienced Cobol programmer can flick through a printout and quickly understand the program. Computer Science students read the C code in the Unix kernel, but seldom change it. They read it so they can read C code easily. We should be able to understand code written by others, not by contraining what others can write, but by more experience in reading what others have written.

But one programmer can only really read code easily in one or two computer languages, just as most people can only learn one or two natural languages really well. For this reason, programmers are categorized by the language they program in.

The Groovy Language will solve this problem by being available for every available AST. As well as the (J)Groovy flavor for the Groovy/JVM AST, the GroovyScript flavor for the Scala parse tree, and the Groovy-DLR flavor for Microsoft's DLR, the Groovy Language will eventually be available for every AST platform. The original (J)Groovy syntax was a close copy of Java's, while being semantically different; Java's was a close copy of C++'s, also semantically different. So someone who knew Java had a head start learning Groovy, and so on.

The Groovy Language will be the end-of-the-line for the C-syntax, available for every practical AST, and so replacing other programmng language syntaxes. Therefore, if someone learns (J)Groovy, they can then switch to using Groovy-DLR easily, just as when someone learns British English, they can switch to using Indian English easily. There will be one Groovy Language to rule other computer language syntaxes. I guess they'll eventually become obsolete.

27 July 2009

What_makes_Scala_groovy?

Lately, I've been thinking about what makes Scala groovier than (J)Groovy...

(1) Nested classes and packages. I can nest my class definitions any way I want when doodling, i.e. doing experimental programming. Groovy DevCon 5 talked about nested classes for Groovy 1.7. Anonymous inner classes aren't necessary, though, as closures can simulate them.

(2) Pattern matching. Pattern matching is an incremental addition to a programming language that, once learnt, is hard to do without. Groovy 2.0 is slated to bring pattern matching, but some think the coming Groovy 2.0 is a myth, just like JSR 241 and the Groovy language spec.

(3) Combinator parsing. People are bored with the limits of regexes, and want more declarative power in parsing stuff. Scala now has a terse combinator parsing syntax, and Scala 2.8 will introduce the more efficient packrat parsing trait for them.

(4) Consistency of syntax and semantics, e.g. the method/field uniform access principle, as opposed to the tack-on approach of (J)Groovy, which is necessary to ensure seemless backwards-compatibility with Java classes. Returning to Groovy coding after working with Scala's "clean break with Java" design, though, is harder than returning to Java coding after working with Groovy. Scala's operator/method and parameter/indexing dualities are features that could successfully be put into Groovy, though.

(5) Mixins/traits. The (J)Groovy 1.1 (betas) AST had empty stubs for mixins, but the Groovy developers never implemented them. After seeing how Scala traits could do things I thought were only elegant with aspects, e.g. the Observer pattern, I now believe Groovy needs those mixins.

(6) Inferred static typing. After working with this in an IDE, one wonders how the "more tests are better than more typing" lie spread so rapidly. Inferred static typing is "more typing (static) with less typing (fingers on keyboard)".

What (J)Groovy features are groovier than Scala's?

(1) Builders. I've heard Ruby copied this feature from Groovy. Has it been done in Scala? Scala's syntax already allows it, though I've yet to see a Scala implementation of Groovy's HtmlBuilder. Scala's inline XML syntax is ugly compared to builder-based syntax.

(2) GStrings. Also known as interpolated strings, they enable us to do much commonly used string handling, e.g. printing, with a terser syntax. Perhaps Scala's scalable syntax could enable these without syntax changes, I don't know.

(3) Dynamic typing. Dynamic typing is useful in the 20% of code where static typing isn't suitable. Just as dynamic Python enhances static C code, dynamic Groovy enhances static Java code. Static languages can emulate some dynamic typing features by typing everything with the Object type, or using an expando object. Open classes that enable inheritance may be impossible to emulate, though. (Groovy also enables built-in static typing which is slower than its dynamic typing. Use this feature for interface documentation only, use Java instead for other static typing requirements.)

It seems Scala could copy (J)Groovy's groovy features far easier than Groovy could copy Scala's. Although Groovy pitches itself as "complementing, not competing with" Scala, since programming in Scala, I've yet to find much that Groovy's a more obvious fit for. Scala's tersity and inferred typing are addictive.

What could make both Scala and (J)Groovy groovier?

Self-mutating syntax. This would enable syntactic macros and keyword aliasing, thus putting Scala into the realm of Lisp/Scheme. Some of Scala's syntax looks like it could be redefined as a syntactic macro, e.g. the for comprehension could generate the underlying calls to map, filter, etc. After pulling out these types of simplifications, perhaps Scala's remaining syntax would be easily handled by a library based on Scala's own parser combinators, making Scala syntax self-referential.

The GroovyScript source code I've posted enables annotations to define lexical and syntactic features of a C-syntax language such as (J)Groovy or Scala, perhaps another way of making it self-referential. It requires using monadic bind and return/value parser combinators, making it a "context-sensitive" grammar. Packrat parsing can do context-free parsing in linear time, though with the cost of memory space: can multicores keep such context-sensitive parsing tractable as well?

22 July 2009

Scala eclipses (J)Groovy

I've posted beta-2 of GroovyScriptonline. GStrings are now parsing. The parser, written in Scala, uses a pushback lexer so lexical tokens can be defined in the parsed syntax using annotations. What's there may be useful for someone to see how a parser with a pushback lexer can work. However, I'm now looking at whether I can rewrite it as an extension to Scala's built-in combinator parser library, so don't expect anything more for a while.

The more I program in Scala, the more convinced I become that it's the grooviest Groovy Language of all. Scala's lexing and syntax needs to be more customizable, though, which is what GroovyScript's all about, adding an alternative lexer/parser to the Scala parse tree, to make the syntax self-referential, thus enabling syntactic macros and keyword aliasing. So a few weeks ago, I decided to switch the primary reference implementation for the Groovy Language from (J)Groovy to GroovyScript. I've still got a lot of learning and work to do though. Just as I programmed in Groovy for a year before surfacing on their mailing list, it will probably take that long or longer before I have much to contribute to Scala.

9 July 2009

The grooviest Groovy of all!

I've posted beta-1 of GroovyScript online. It's a lexer and parser with just-in-time pushback lexing, with an Apache licence, written in Scala. When the parser backtracks, it pushes unused tokens back into the lexer. Hence we can write a lexer/parser that enables lexical definitions to be defined as annotations using regexes in the parsed code. The following code snippet parses correctly in beta-1:

abc;
@AddComment('//[^\r\n]*') try{
defg; //hi!!!
987;
@Anno try{
@DoIt(7, 'abc',) zyx;
16.8
};
'bcdefg';
};
@Anno @Letter hijk;
lmnop;

The @AddComment annotation enables //-comments to be recognized as whitespace within its tagged statement, and eventually all external files parsed from within, but not before or after the tagged statement. That's all that's working for now, but I'll eventually put in everything I blogged about in my last post, e.g. custom lexing rules, syntactic macros, name aliasing, a Unicode IME. Scala certainly proved its worth for this challenging exercise, making me think about the best path forward for GroovyScript, the 3rd language in the Groovy Programming Language family, after (J)Groovy and Groovy-DLR. As a result, I'm switching the GroovyScript target platform from the (J)Groovy AST to the Scala 2.8 parse tree.

Why? It happened like this... About a year ago, I started converting some hard-to-debug lexer/parser code written in Groovy to C#, just to code it anew somewhere, hoping to debug the logic. The Visual Studio editor complained about the static types not matching. I fiddled it so the types matched, then discovered I had also debugged the logic problem. That was when I started to reconsider the supposed benefits of dynamically typed languages. Static typing Java-style is verbose, though, but with type inference it rocks! C# has some type inference, but Scala's is incredible! I now seriously doubt the benefits of dynamic typing over its costs.

I started off building Groovier for the GrAST in Scala, but discovered Scala itself was already the grooviest language of all! First came C, then C++, then Java which should be called C3+, because after that C# came along, the sharp symbol (#) being 4 plus signs (+) joined together, which then makes Scala be C5+. Because I'm now more impressed with the Scala language engine than with (J)Groovy's, I've decided to switch GroovyScript's target platform from the GrAST to the Scala 2.8 parse tree. With inferred static typing, it's at a higher level of abstraction than the GrAST. And unlike the GrAST, I can bundle it with GroovyScript because its name is different.

But not only that, Scala's also multi-platform, running on both the JVM and the CLR. Are Codehaus (J)Groovy or Codeplex Groovy-DLR really needed? So as the underwriter for the Groovy Language, I'm also switching from (J)Groovy to GroovyScript as the primary reference implementation for the Groovy Language. The Scala language engine is now the primary platform for the Groovy Language. (J)Groovy was the first language in the Groovy Language family, but GroovyScript will soon be the leading-edge one. GroovyScript will then change its name to Groovy 2.0. Because it's now Apache-licensed, the (J)Groovy developers could adapt it to the GrAST and bundle it with (J)Groovy if they really wanted to.

16 June 2009

Gr8 isn't great, it grates

A few short months after my very first posting to the Groovy Language mailing list, Graeme Rocher changed the name of Groovy on Rails to Grails. It was too late to change Groovy's groovy name as well, but I suspect the Groovy developers will do so by stealth for version 2. As the underwriter of the Groovy Language, I must ensure Groovy's development continues, and do so under its present name.

The Groovy developers recently created the "Gr8 family of technologies" brand, i.e. Groovy / Grails / Griffon / Gant / etc (see http://twitter.com/aalmiray/status/1906155191) in direct response to my blog post at http://gavingrover.blogspot.com/2008/11/groovy-language-family.html. I suspect "Gr8" is also their upcoming name for the dynamic language engine inside Groovy 2.0, to compete with Google's V8 engine inside Chrome JavaScript. Of course, the "Gr8 dynamic language engine" would soon after become an engine for all JVM-based dynamic programming languages, itself a good idea, but the SpringSource developers might then quietly ignore Groovy 2.0 support in favor of other languages running on the Gr8 engine. Like the 5 yr old JSR at http://www.jcp.org/en/jsr/detail?id=241, Groovy 2.0 would become a carcass, its only purpose to prevent anyone else using the brand.

GroovyScript will be a GPL-licensed lexer/parser for the language engine inside the Groovy 2.0 Language. If that engine changes its name, GroovyScript will then be allowed to bundle the engine as part of its distro.

Orders of Infinity

2010-04-30T14:59:00.000-07:00

Originally posted 14 April 2010 on my temporary blogspace...

Eighteen months ago I posted a blog entry describing a Mass-Parity-Distance-invariant Universe. I got that idea while reading Roger Penrose's The Road to Reality during a month of vacation time away from online and other distractions. I couldn't see anything in the book that contradicted the idea. I knew String Theory is the most popular theory on how to unite the four forces, explain the Universe, and all that, but whenever I read up about it I sensed a certain inelegance about it all. Because I knew Penrose felt the same way, I was keen to plow through his book when I had the time. I haven't read up on physics much since writing that blog, but it's hard not to think about the maths when I'm out for a walk and daydreaming, and so I've had some further ideas for the maths...

Physics recap

I explained how by using a Mass-Parity-Distance (MPD) symmetry, similar to the well-known Charge-Parity-Time (CPT) symmetry, we would have a Universe where equal amounts of positive and negative energy fly off in opposite directions at the big bang, each side self-attracting but mutually repelling. At the Big Bang, the left-handed gravitons and left-handed neutrinos fly off in one direction, thus equating positive mass with normal matter in the large, while their right-handed counterparts would fly off in the other direction, equating negative mass with anti-matter. Matter and antimatter created from positive energy in our side of the Universe would both have positive mass, but a virtual particle-antiparticle pair in a vacuum would have an overall energy of zero, one of the pair having positive energy, the other, negative.

I also explained how such an MPD-symmetric Universe could explain dark energy, though I suspect for it to have its observed strength and timing, the observable Universe would be a very tiny proportion of the actual Universe. Just as our sun is one of about 100 billion in the Milky Way, and our galaxy one of about 100 billion in the observable Universe, so our observable Universe could also be a 100-billionth of the actual Universe. Homogeneity and isotropy would be a more local effect to this scenario. To picture it all using the common 2D-curved-space picture for general relativity, the positive matter would be on top of the sheet sinking downwards, but the negative matter would be under the sheet to indicate negative distances, floating upwards to indicate the negative mass. The positive and negative matter would both self-gravitate, but repel the other.

Cantor's hierarchy of infinities

I suggested the dual CPT and MPD symmetries suggest a mathematical model where the Universe is made up of two complex planes rather than four real lines. A complex plane is different to a 2-real-D plane in not having reflectional symmetry about the real line: two complex planes would have two such assymmetries, perhaps explaining the "parity" in both the CPT and MPD symmetries. But to see why our Universe would be two complex planes related in some way instead of some other structure, we'd need to understand what would make such a structure special. Why two complex planes and not three? And why use complex planes instead of real lines? What might make such a structure special could be the same thing that makes a single complex plane special. I suggest we look at its position in Cantor's hierarchy of infinities, as that seems more foundational than the other branches of mathematics, and move on up from there...

We know the zeroith order of infinity (a.k.a. finiteness) can define a logic system, by using two or more ordered finite values, i.e. false and true for boolean logic, more for other logic systems. The first order of infinity (a.k.a. aleph-0) can define an arithmetic system. The simplest is the natural numbers, created by starting at number 1 and applying induction. Extensions to this such as the integers and rational numbers are also aleph-0. Godel showed this arithmetic system cannot be both consistent and complete.

Things get more interesting at the second order of infinity. A next higher order of infinity is the power set of a lower one. The real numbers, introducing the square root of 2, extend the integers. We can prove they're at some higher order than the first order (integers, etc), but can't prove at what order they are. We therefore speculate they're at the second order (a.k.a. aleph-1), calling this the Continuum Hypothesis. Other number systems with the property of continuity (e.g. complex numbers, n-D manifolds) would then also be aleph-1, but complex numbers, which introduce the square root of -1, are "algebraically complete", not requiring any further extensions.

And what of aleph-2, the third order of infinity? The set of all curves (including fractal ones) is known to be at some order of infinity above aleph-1, hypothesized to be aleph-2. When looking at the curves, maybe it's best to consider the most complex curves first, such as 1D-lines with a (Hausdorff-) dimension of 2. The most famous of these are the Mandelbrot and Julia sets, which both happen to be defined on the complex plane. When we look at computer simulations of them, we see many concentric closed curves of various colors, reflecting the different integral-valued accuracies of calculation. If we could mathematically define a real-valued accuracy of calculation, would we still see concentric fractal curves? I suspect so, and that they would merge into a continuously-varying fractal structure with (Hausdorff-) dimension of 3, on the 2-dimensional complex plane. All of the Julia sets for the standard Mandelbrot look like they have this same concentricity property, though only some of them seem to have (Hausdorff-) dimension 3, the rest (on inspection) seeming to have dimension of less than 2. One is even a perfect circle, with only (Hausdorff-) dimension 1. And of course we could consider the nonstandard Mandelbrot views for all these Julia sets.

The Julibrot Set

Let's use these two fractals in a candidate mathematical model for our Universe. The Julibrot Set is defined as the topologically-4D union of all the Julia sets of the Mandelbrot set. If we consider real-valued accuracies of calculation for the entire Julibrot, we have a (Hausdorff-) dimensionality of somewhere between 4 and 6, embedded in the 4 topological dimensions. Now I suspect there could be a theorem concerning the (Hausdorff-) dimension in such a structure: I'll speculate it's 5 and see where that leads. If such a 2-complex-plane structure explains the 4D-spacetime of our Universe, then there's an extra non-expansive dimension supplied by considering the fractal curves at different positive-real-valued degrees of accuracy. This would explain the phenomenon of mass in our 4D-spacetime, along with certain rules of distribution within, which could be the law of Einsteinian gravity.

But where would this positive real number come from? In my other blog entry, I suggested we could relate each complex plane to the other probabilistically! Each position in a Julia set would correspond only probabilistically to the positions in the associated nonstandard Mandelbrots. This would explain why the said mass in our Universe doesn't conform fully deterministically to the large-scale gravity-based rules of distribution, but has degrees of freedom as ultimately enabled by the law of quantum physics, as powered by Planck's constant, being the measure of probability relating the two complex planes together. Such a structure consisting of 2 complex planes related probabilistically therefore would be the minimumly complexed structure that can squeeze in a fifth real-valued dimension, the one we know as Mass, into the four expansive dimensions.

What would be the order of infinity for this Julibrot-based Universe? There could be a theorem saying this structure is necessary and sufficient to contain all curves, and is therefore at the third order of infinity, aleph-2. Our Universe could then be the minimumly-complexed structure that can exist at aleph-2. Furthermore, just as the complex numbers are algebraically complete at aleph-1-infinity, so also our 4D-spacetime with its inbuilt phenomenon of Mass, obeying rules both of gravitation and of quantum physics, could also be complete in some way at aleph-2-infinity.

The directed dimension and inbuilt polarity

Is this probabilistically-defined Julibrot set the best model for our Universe? Any recursively-applied polynomial equation seems to give the basic Mandelbrotly-edged shape on the computer, so presumably they all have high enough Hausdorff dimension to be a candidate model. But only simple Julibrot Sets are reflectionally or rotationally symmetric in 3 dimensions, but not in 4, giving a dimension that looks like Time.

By looking at the Julibrot's constituent Mandelbrot and Julia sets on the computer, including their colored accuracy levels, we see that the Mandelbrot-real dimension is the assymmetric dimension, i.e. the Time dimension. As a bonus, the Mandelbrot-imaginary dimension is the only one that's reflectionally symmetric, therefore the one along which positive and negative matter flew apart, i.e. the Axial Space dimension. The Julia sets at each point below the standard Mandelbrot Time axis (where y=0 on the plane) are similar to the corresponding one above, except for being reflected through their Julia-real axis. This gives the appearance of the Mass rotating in opposite directions in each half of the structure, perhaps suggesting the opposite handedness of gravitons and neutrinos in each half of the mass distribution in our Universe. But in each half of the Mandelbrot Mass distribution, when we look toward the Time axis, the Mass appears to rotate in the same direction, perhaps suggesting why space appears to have an inbuilt polarity.

The Julibrot generated by the standard parameter plane doesn't bear an obvious visual resemblance to the MPD-symmetric Universe I've been describing, but by using a non-standard parameter plane for the Mandelbrot, we can shape it more like our Universe, such as the 1/μ-plane for a Big Crunch universe, or the 1/(μ + 0.25) plane for a Heat Death one, as pictured on this page.

We can also see suggestions of increasing entropy of Mass by looking at the Mandelbrot set. Because of the second law of thermodynamics, the entropy of the Universe at the Big Bang was at its minimum. In a Big Crunch Universe, the likely death scenario in both the positive matter and negative matter sides of the Universe is as two black holes spinning around each other for trillions of years before eventually smashing together, a high entropy end-state. We can consider the directedness of the dimension of Time as being caused by the increasing entropy of the Mass inside the Universe along the Time dimension. The standard Mandelbrot set on various parameter planes (i.e. the Time and Axial Space dimensions) seem to have one "creation" point, where the (Hausdorff-) dimensionality is 1 just at that point, situated on the Time axis, suggesting the low-entropy Big Bang.

So we could say that by increasing the mapping uncertainty between the Mandelbrot and Julia sets of complex planes (i.e. Planck's constant) up from zero, we create a fifth, non-expansive, dimension called Mass, which causes the Mandelbrot-real dimension to be directed and become Time, and the Mandelbrot-imaginary dimension to become a spatial axis around which the Julia planes rotate.

Electromagnetism

In this probabilistic Julibrot model I've described, Charge is seemingly absent. We first considered this type of model because of the CPT and MPD dual symmetries suggesting two complex planes, but have only derived out the entity of MPD-symmetric Mass following certain gravitational and quantum rules. We can consider the forces without an infinite range (the strong and weak forces) to be more minor details for filling in later, but can't consider electromagnetism that way.

We can put electromagnetism into this model by introducing an additional relationship into the structure, besides the probabilistic one between the two complex planes. The basic apparent difference between MPD-symmetric gravity and CPT-symmetric electromagnetism is that with gravity, like masses attract while unlike ones repel, whereas with electromagnetism, like charges repel while unlike ones attract. The logical effect of this is that gravity's masses are real numbers, enabling aggregation of mass, while charges must be discrete. These look like big differences, but there's a simple structural property we can introduce into the model which makes gravity and electromagnetism be exactly the same force!

Someone once asked me how we know the Universe isn't like at the end of the Men in Black: 1 movie, where it's all just a speck of dust on another creature's back. I've since realized that unless the Universe looks exactly the same at its largest scale as it does at its smallest, then it's not really a self-contained system. The generally-agreed smallest scale of the Universe is a particle and anti-particle splitting apart, then perhaps coming back together. In an MPD-symmetric Universe, the largest scale is positive matter and negative matter splitting apart in opposite directions along one dimension of space, then, in some cosmological theories, coming back together again. If the very first particles to split apart at the Big Bang were some form of particle/graviton coupling, splitting away from their opposite forms, then the negative mass will be mainly antimatter, just as the positive mass we observe around us is mainly normal matter. And a virtual particle and anti-particle splitting apart and coming back together would have matching positive mass and negative mass.

So, the only difference between gravity and electromagnetism in such a MPD-symmetric neutrino/graviton-origined Universe is that gravity is what we see when we're on the inside looking outwards, and electromagnetism is what we see when we're on the outside looking inwards. On the inside looking outwards, there's only one instance to look at, but on the outside looking in, we see many instances. From the inside looking out, it looks like MPD-symmetric positive and negative Mass obeying the laws of gravity, but from the outside looking in, it looks like CPT-symmetric positive and negative Charge obeying electromagnetic laws.

Black holes could fit naturally into this model. Some people speculate they're inherently similar to particles, others say they're the outside of other Universes. And when we look at a computer simulation of the Mandelbrot set, we see many near-similar instances of the large-scale Mandelbrot at smaller and smaller scales, ditto with the Julia sets, and perhaps this mathematical fact suggests in some way this physical fact about the Universe, that the Universe must always look the same at its largest scale as it does at its smallest. And in fact, this mathematical fact suggests perhaps we don't need to introduce this physical fact as an additional relationship into our structure, but perhaps it falls naturally out of it.

At the Big Bang

What might the Universe look like under this model in the first instant after the Big Bang? Because the neutrino is the particle defining matter/antimatter, I'll call the first particle to split from its antiparticle at the Big Bang a proto-neutrino. When the first protoneutrino/graviton coupling split apart from its opposite at the Big Bang, but before either the protoneutrino or the protoneutrino decayed into more particles, each half of the Universe looked like an expanding outer balloon with one decaying inner balloon inside it. Both outer and inner balloons had the same inherent structure: from the viewpoint of the space between them, the outer balloon appeared to obey the laws of MPD-symmetric Einsteinian gravity, and the inner balloon appeared to obey the laws of quantum electrodynamics. As the protoneutrino decayed into further different particles, the outermost scale and the innermost (quantum) scale always kept the same mathematical structure.

The protoneutrino further decayed into other particles, and in our own positive matter side of the Universe, in our own observable portion of the Universe, the fermions are presently the our well-known leptons, quarks, weak-force bosons, and whatever dark matter particles there are. In unobservable portions of the positive matter side of the Universe, perhaps the fermions are different. Perhaps the masses of particles change depending on when and where they are in the Universe. Perhaps other dimensionless constants, such as the fine structure constant, also change. In the negative matter side of the Universe, perhaps the first few quantum rolls of the dice caused the anti-protoneutrino to decay differently, resulting in a totally different life story there, but the overall mass distribution would be the same as in the positive matter side of the Universe, and there would also be a direct identity lineage from the protoneutrino to our present-day neutrino, ditto in the negative matter Universe. But despite what other changes there are in the structure of the Mass, the smallest scale would always have the same structure as the largest scale, and any changes to the structure of the Mass, such as values of particle masses or the fine structure constant, would actually be caused by this self-similarity requirement.

Higher orders of infinity?

So we've seen a possible model of our physical Universe by regarding Mathematics not as something that simply describes the Universe, but as something which the Universe is at some order of infinity. I suspect if my conjectures above are correct, then the core theorem deriving the axioms of General and Special relativity and quantum electrodynamics from an uncertainty-based Julibrot set would be as significant at aleph-2-infinity as the Cauchy-Riemann theorem is at aleph-1-infinity.

If our Universe is what exists at the third order of infinity, then what might constitute the next higher order of infinity? Using Cantor's theorem, by considering the power set of our Universe, we might say it's the set of everything that could have happened, could yet happen, and would yet have happened in our Universe, except for the number of dimensions. But how would we place our own specific Universe of actual happenings in this picture, with its unique quantum reductions into actualities? One person might say the power set is at a higher order of infinity than our own specific Universe of actual happenings, because of Cantor's theorem. But someone else might say the Universe of happenings is at a higher order, being the real, intentioned actualization, while the possibilities are at a lower order, being just the canvas for the actualization, so to speak. It sounds like an argument between atheists and theists, and perhaps never able to be proven either way.

So possibly our own consciousnesses can't really comprehend very high up the ladders of infinity. At the first order, we can't logically define a consistent and complete system. At the next order, we must hypothesize its actuality as The Continuum in our own Universe. And at another order or two higher, where our own consciousnesses dwell, we can't logically prove which of the two orders is the higher and which is the lower. If we could look higher up the hierarchy of infinities to the infinite level, then the hierarchy itself would be able to be counted by an induction-based counting scheme, and so become self-referenced, and even itself an inconsistent and incomplete arithmetic system, which it isn't. In fact, because different instances of our human consciousnesses can't agree which of the third and fourth orders of infinity is higher than the other, we can't even make the known physical representations of the orders of infinity into a propositional logic system, not knowing which to call True.

Moving each order of infinity up the ladder seems to require utilizing some new well-known mathematical concept. Moving from finiteness to aleph-0 requires Induction, and from aleph-0 to aleph-1 requires Continuity. If an MPD-invariant Universe is at aleph-2, then moving there requires Probability. This would be the lowest order of infinity which contains the entities of Time and Mass, as distinct from Space. And such entities, Time and Mass, are required for the mathematics of computational complexity, Time as a resource and Mass for building Turing machines. Perhaps the next order of infinity, aleph-3, requires the concept of Computation, which could explain the phenomena of consciousness. Such Computation is also required for calculating fractal curves in aleph-2 spacetime.

Computational complexity

Perhaps computational complexity will play an important role in a theory of the Universe. There's now many known complexity classes in the zoo. If we can slot the theory of computational complexity directly onto a mathematical structure which defines a space-distinct Time, which behaves according to the laws of our Universe, then many of these complexity classes, such as PSPACE and P(TIME), may meld into one when we factor in the effects of Special and General Relativity, such as time dilation and space contraction. PSPACE problems require more computation "power" than P(TIME) problems, but if space can become time due to high acceleration or a nearby strong gravitational field, perhaps ultimately they're really the same complexity class.

Similarly, the distinction between the P(TIME) and NP(TIME) complexity classes may not exist at Planck scales because of the quantum nondeterminism. Perhaps the electric field generated by human brain neural structure makes use of such quantum nondeterminism to produce the effect of consciousness. Perhaps both large-scale (relativistic) and small-scale (quantum) effects together reduce the many complexity classes down to a mere few. They seem to fall into four broad groups: logarithmic, polynomial, exponential, and recursive.

Do these complexity groups each match up somehow to the various orders of infinity I've described? Do aleph-0-infinite structures like the integers relate somehow to logarithmic-space computation? Do aleph-1-infinite structures like the complex numbers relate to polynomial-resource computation? Is our Universe an aleph-2-infinite structure? Is it an MPD-invariant "canvas" for our Universe of actual happenings, related somehow to exponential-resource computation? Are the quantum reductions that form the actual happenings in our Universe, including our own consciousnesses, an aleph-3-infinite structure? Related somehow to recursively-enumerable computation? And what could possibly lie beyond that?

Programming Language Structure

2010-04-30T14:45:00.000-07:00

Originally posted 16 January 2010 on my temporary blogspace...

Programming languages have their origin in natural language, so to understand the structure of computer languages, we need to understand natural ones. According to Systemic Functional Grammar (SFG) theory, to understand the structure of language, we need to consider its use: language is as it is because of the functions it's required to serve. Much analysis of the English language has been performed using these principles, but I haven't found much on programming languages.

Functional grammar of natural languages

According M.A.K. Halliday's SFG, the vast numbers of options for meaning potential embodied in language combine into three relatively independent components, and each of these components correspond to a certain basic function of language. Within each component, the networks of options are closely interconnected, while between components, the connections are few. He identifies the "representational" and "interactional" functions of language, and a third, the "textual" function, which is instrumental to the other two, linking with them, with itself, and with features of the situation in which it's used.

To understand these three components in natural languages, we need to understand the stages of encoding. Two principle encodings occur when speech is produced: the first converts semantic concepts into a lexical-syntactic encoding; the second converts this into spoken sounds. A secondary encoding converts some semantics directly into the vocal system, being overlaid onto the output of the lexical-syntactic encoding. Programming languages have the same three-level encoding: at the top is the semantic, in the middle is the language syntax, and at the bottom are the lexical tokens.

The representational function of language involves encoding our experience of the outside world, and of our own consciousness. It's often encoded in as neutral a way as possible for example's sake: "The Groovy Language was first officially announced by James Strachan on Friday 29 August 2003, causing some to rejoice and others to tremble."

We can analyze this as two related processes. The first has actor "James Strachan", process "to officially announce", goal "the Groovy Language", instance circumstance "first", and temporal circumstance "Friday 29 August 2008"; the second process is related as an effect in a cause-and-effect relationship, being two further equally conjoined processes: one with process "to rejoice" and actor "some"; the other with process "to tremble" and actor "others".

The interactional function of language involves injecting the language participants into the encoding. A contrived example showing many types of injects: "The Groovy Language was first announced by, of all people, creator James Strachan, sometime in August 2003. Was it on Friday 29th? Could you tell me if it was? Must have been. That august August day made some happy chappies like me rejoice, didn't it?, yeehaaaah, and probably some other unfortunates to tuh-rem-ble, ha-haaah!"

We see an informal tone, implying the relationship between speaker and listener. There's glosses added, i.e. "of all people", "august", "happy chappies like me", "unfortunates", semantic words added, i.e. "creator", semantic words removed, i.e. "officially", sounds inserted, i.e. "yeehaaaah", "ha-haaah", prepended expressions of politeness, i.e. "Could you tell me if", and words spoken differently, e.g. "tuh-rem-ble". Mood is added, i.e. a sequence of (indicative, interrogative, indicative). Probability modality is added, i.e. "must have", "probably". We could have added other modality, such as obligation, permission, or ability. We've added a tag, i.e. "didn't it?". We could have added polarity in the main predicate. What we can't indicate in this written encoding of speech is the attitudinal intonation overlaid onto each clause, of which English has hundreds. Neither can we show the body language, also part of the interactional function of speech.

Natural language in the human brain

A recent article in Scientific American says biologists now believe the specialization of the human brain’s two cerebral hemispheres was already in place when vertebrates arose 500 million years ago, and that "the left hemisphere originally seems to have focused in general on controlling well-established patterns of behavior; the right specialized in detecting and responding to unexpected stimuli. Both speech and right-handedness may have evolved from a specialization for the control of routine behavior. Face recognition and the processing of spatial relations may trace their heritage to a need to sense predators quickly."

I suspect the representational function of language is that which is produced by the left hemisphere of the brain, and the interactional function by the right hemisphere. Because the right side of the brain is responsible for unexpected stimuli, from both friend and foe, then perhaps interactional language in vertebrates began as body language and facial expressions to denote conditions relevant to others, e.g. anger, fear, affection, humidity, rain, danger, etc. Later, vocal sounds arose as the voice box developed in various species, and in humans, increasingly complex sounds became possible. The left side of the brain is responsible for dealing with regular behavior, and so allowed people to use their right hand to make sign language to communicate. Chimpanzees and gorillas use their right hands to communicate with each other, often in gestures that also incorporate the head and mouth. The article hypothesizes that the evolution of the syllable in humans triggered the ability to form sentences describing processes involving people, things, places, times, etc. Proto-representational language was probably a series of one-syllable sounds similar to what some chimps can do nowadays with sign language, e.g. "Cat eat son night". Later, these two separate functions of natural language intertwined onto human speech.

Programming language structure

When looking at programming languages, we can see the representational function easily. It maps closely to that for natural languages. The process is like a function, and the actor, goal, recipient, and other entities in the transitive structure of natural language are like the function parameters. In the object-oriented paradigm, one entity, the actor, is like the object. The circumstances are the surrounding static scope, and the relationships between processes is the sequencing of statements. Of course, the semantic domains of natural and programming languages are different: natural languages talk about a wider variety of things, themselves more vague, than programming languages. But the encoding systems are similar: the functional and object-oriented paradigms became popular for programming because between them it's easy for programmers to code about certain aspects of things they use natural language to talk about. The example in pseudocode:

Date("2003-8-29").events += {
  def a = new Instances();
  a[1] = jamesStrachan.officiallyAnnounce(Language.GROOVY);
  a[1].effect = [some: s => s.rejoice(), others: o => o.tremble];
}

The similarities between the interactional functions of natural and programming languages is more difficult to comprehend. The major complication is the extra participants in programming languages. In natural language, one person speaks, maybe one, maybe more people listen, perhaps immediately, perhaps later. Occasionally it's intended someone overhears. In programming languages, one person writes. The computer reads, but good programming practice is that other human people read the code later. Commenting, use of whitespace, and variable naming partly enable this interactional function. So does including test scripts with code. Java/C#-style exception-handling enables programmer-to-programmer interaction similar to the probability-modality of English verbal phrases, e.g. will/definitely, should/probably, might/could/possibly, won't, probably won't.

Many programming systems allow some interactional code to be separated from the representational code. One way is using system-wide aspects. A security aspect will control the pathway between various humans and different functions of the program while it's running. Aspects can control communication between the running program and different facets of the computer equipment, e.g. a logging aspect comes between the program and recording medium, a persistence aspect between the program and some storage mechanism, an execution performance aspect between the program and CPU, a concurrency aspect between the program and many CPU's, a distribution aspect between the program and another executing somewhere else. Here, we are considering these differents facets of the computer equipment to be participants in the communication, just like the programmer. Aspects can also split out code for I/O actions and the program entry point, which are program-to-human interactions. This can also be done by monads in "pure functional" languages like Haskell. Representational function in Haskell is always kept separate from interactional functions like I/O and program entry, with monads enabling the intertwining between them. Monads also control all access between the program and modifiable state in the computer, another example of an interactional function.

Textual function of language

The textual function of language in SFG is that which concerns the language medium itself. In spoken natural language, this is primarily the sequential nature of voice, and in written language, the 2-D form of the page. Whereas in natural language theory, the voice-carrying atmosphere and the ink-carrying paper are obviously mediums and not participants, it's more difficult to categorize the difference between them in programming language theory. Because a program is written as much for the CPU as for other human readers, if not more so, we could call the CPU a participant. But then why can't the CPU cache, computer memory, hard-disk storage, and comms lines also be called participants? Perhaps the participants and the transmission medium for natural languages are also more similar than different.

The textual function of language is made up of the thematic, informational, and cohesive structures. Although mainly medium-oriented, they also involve the participants. The thematic structure is speaker-oriented, the informational structure is listener-oriented. The thematic structure is overlaid onto the clause. In English, what the speaker regards as the heading to what they're saying, the theme, is put in first position. Not only clauses, but also sentences, speech acts, written paragraphs, spoken discourses, and even entire novels have themes. Some examples using lexical items James, to give, programmers, Groovy, and 2003, with theme in italics:

James Strachan gave programmers Groovy in 2003.
Programmers are who James gave Groovy to in 2003.
The Groovy Language is what James gave programmers in 2003.
2003 is when James gave programmers Groovy.
Given was Groovy by James to programmers in 2003.

In English, the Actor of the representational function's transitive structure is most likely to be separated from the interactional function's Subject and from the Theme in a clause, than those from each other. I think the textual functions of natural language are far more closely linked to the interactional function than to the representational. Perhaps the right side of the brain also processes for such texture structure.

The informational structure jumps from the top (i.e. semantic) encoding level directly to the bottom (i.e. phonological) one in English, skipping the middle (i.e. lexical/syntactic) level. This is mirrored by how programming languages such as Python use the lexical tokens to directly determine semantic meaning. In English, the speech is broken into tone units, separated by short pauses. Each tone unit has the stress on some part of it to indicate the new information. For example, each of these sentences has a different informational meaning (the bold indicates the stresses):

James gave programmers Groovy in 2003.
James gave programmers the Groovy Language in 2003.
James gave programmers Groovy in 2003.
James gave programmers Groovy in 2003.
James Strachan gave programmers Groovy in 2003.

Unlike the thematic structure, the informational structures the tone unit by relating it to what has gone before, reflecting what the speaker assumes is the status of the information in the mind of the listener. The informational structure usually uses the same structure used in the thematic, but needn't. English grammar allows the lexical items to be arranged in any order to enable them to be broken up in any combination into tone units. For example, these examples restructure the clause so it can be divided into two tone units (shown by the comma), each with its own stress, so two items of new information can be introduced in one clause:

James gave Groovy to programmers, in 2003.
As for Groovy, James gave it to programmers in 2003.
In 2003, James gave programmers Groovy.

Programming languages should follow the example of natural languages, and allow developers to structure their code to show both thematic and informational structure. The final textual function, the cohesive structure enables links between clauses, using various techniques, such as reference, pronouns, and conjunctions. Imperative programming languages rely heavily on reference, i.e. temporary variables, but don't use pronouns very much. Programming languages should also provide developers with many pronouns.

Summary

Programming languages initially represented information in the same way humans do, using transitive structures such as function calls, joined by logical relationships such as blocks and class definitions. Interactional aspects of code were initially intertwined, but could be separated out using aspects and monads. Enabling different textual structures in programs isn't very widespread, so far limited to providing different views of an AST in an IDE, only occasionally allowing "more than one way to do things" at the lexical level. When used well, textual structures in code enable someone later on to more easily read and understand the program.

In promoting the benefits of programming languages enabling different textual structures, I think it's useful to narrow down to two primary structures: the transitive and the thematic, as these two are easiest to communicate to programmers. See my earlier thoughts on how a programming language can enable more thematic variation. Programming languages of the future should provide the same functions for programmers that natural languages provide for humans.

And of course, I'm building Groovy 2.0, which will both enable thematic variation in the language syntax/morphology, and supply a vast vocabulary of Unicode tokens for names. The first iteraction will use Groovy 1.x's SwingBuilder? and ASTBuilder, along with my own Scala-based combinator parsers, to turn Groovy 2.0 source into Groovy 1.x bytecode. The accompanying Strach IME will enable programmers to enter the Unicode tokens intuitively. Groovy 2.0 will break the chains of the the Antlr/Eclipse syntactic bottleneck over Groovy 1.x !!!

Groovy's Groovier Roadmap

2009-05-09T16:24:00.000-07:00

I've just put the source code for beta-04 of Groovy-DLR 1.0 online, with some improvements, including evaluating code in separate files, and in-place comment lex-definitions. Time for a brief roadmap...

In-place lexing definitions
Beta-4 features a basic in-place comment lex-definition feature, e.g. addcomment "(?s:#{4}.*)" in the source will cause the parser to add this comment definition to the whitespace lexer for subsequently parsed and/or evaluated files. This particular example of comment lets us add #### in the source file to comment to end-of-file. (We'll change the addcomment keyword into an annotation later.) In-place string lex-defines are also coming. For my earlier thoughts on in-place lexical definitions, see part one and part two.

To make such in-place lex-definitions finer grained, I need to weave just-in-time lexing (with pushbacks) throughout the parser. I've just started on this. We could also define other lex-definitions in the source, e.g. numeric and date formats. See an earlier post of mine for more ideas on this.

In-place parsing hooks
We'll also provide hooks in the parser to enable Groovy programmers to define their own primary expressions, path elements, operators, and statements. Path elements are a more recent innovation in C-syntax languages. We'll change the postfix operators, ++ and --, into path elements. The operators will then have 4 well-defined level groups: right-associative prefix unary, left-associative binary, right-associative ternary, and right-associative assignment.

About a year ago, Larry Wall give his 12th Annual State of the Onion talk, talking about progress on the Perl 6 language. His parser didn't use numbers to define precedence levels, instead used "surreal" precedence, i.e. defined a precedence with respect to an earlier operator definition, specifying either "same as", "looser than", or "higher than". We'll use this technique for the operator hooks.

Implied statement/etc termination
Currently, I do a 2-phase parse on statements to implement the implied semicolons. For the first phase, I grab tokens up to the following newline or semicolon. The second phase attempts to parse them, accepting every line that parses into some valid node. Every line that doesn't parse successfully, we continue the parsing by appending the next group of valid tokens. This means we can end a line with tokens like + or ., but not begin one with these. Later, I'll add a third phase, so for every line that fails to parse before reaching its end, it'll be joined to the end of the previously parsed tokens and parsing resumed from those. I'll then abstract this technique to all lists in Groovy's syntax. I've already blogged about this.

Such multi-phase parses let us do context-sensitive parses. I use a similar technique with the lexer/parser to parse valid GStrings. I use a ManyLazily combinator parser, which itself is composed of Bind and Retn parsers. (Of course, such stuff can become inefficient, and even intractible, very quickly, but we won't worry about that until later. Premature optimization is evil, right? We'll then add memoization, perhaps even experiment with multicore stuff. Parsing is one algorithm that could really benefit from the looming multicore revolution.)

Multi-option syntax forms
Python uses indentation to group statements into blocks, C-syntax languages use curlies. Groovy will provide both techniques simultaneously, using symbol {:. When working, we'll abstract the technique to other lists in the syntax using symbols (: and [:. Groovy developers will have the choice of syntactic forms, even able to mix both styles together in the same portion of code, just like in natural language.

Another code formatting technique related to indenting is spacing on a single line. I often use the number of spaces between tokens to show how items are grouped, to make the code more readable. An example where I use 3 spaces instead of none or one to show groupings on one line:
[Trig.CTD, object.method( call( [7.89, 'abc'], 7 * (2+x), Trig.END ), Parse.jig(99) )]
I use this technique all the time when coding. Just as Groovy will enable both list-bracketing and indentation, so also it will enable both expression-bracketing and token-spacing. So 7 * 2+x will evaluate as 7 * (2+x) due to the spacing. The Fortress language already checks and rejects code where the spacing doesn't match the syntactic bracketing; the Groovy Language will go a step further and actually enable both formats. Every degree of syntactic freedom will be utilized in the Groovy Language syntax, just like in natural language.

Name aliasing
Name aliasing is what got me interested in writing an alternative lexer/parser for the GrAST (Groovy AST) in the first place. I've long wanted to use Chinese characters as aliases for the English names and keywords in programs, to make the code shorter. I don't want to write this:

content.tokenize().groupBy{ it }.
  collect{ ['key':it.key, 'value':it.value.size()] }.
  findAll{ it.value > 1 }.sort{ it.value }.reverse().
  each{ println "${it.key.padLeft( 12 )} : $it.value" }

when I can write this:

物.割().组{它}.集{ ['钥':它.钥, '价':它.价.夵()] }.
  都{它.价>1}.分{它.价}.向().每{打"${它.钥.左(12)}: $它.价"}

(You need a CJK font to see the above correctly.)

Groovy will enable the keywords and names in programs to be aliased lexically. Even right-directional languages will be enabled. All the vocabulary of Unicode will be tightly integrated into the Groovy Language.

Custom input method editor
Because I think CJK characters will prove quite popular when Groovy enables them, I'm also working on an Input Method that non-Chinese like me can learn and use easily. Last year, I posted online a graphical decomposition of the 20,000 most common Chinese characters, both simplified and complex ones. I periodically munge this to get ideas for the best component-to-key assignments, but won't make any decisions until much later. Such key assignments are analogous to natural language phonetics, and can't be changed easily once people start using them, so I'm not releasing anything else too quickly in this area.

An easily-used IME for CJK characters is just the first task; ultimately the IME will enable every Unicode character, even SMP ones, to be entered. Within 10 years, developers could be programming in Egyptian heiroglyphs for fun! Can you see my vision for Groovy, anyone???

One last thing...
Oh yes, I almost forgot to mention, I'm not doing any of this in C#. I'm switching languages, back to the JVM, back to the (J)Groovy AST, to SCALA.

I'd been dabbling a bit with Lisp/Scheme and Haskell lately, and have wanted to learn a functional language thoroughly, but hadn't known which one till now. I chose Scala because:

I can start doing things in the object-oriented style, and incrementally switch to the functional style

I can easily port Groovy-DLR into it, which gives me a real-world programming task to use it for

I can use it to write to the (J)Groovy AST, putting me back onto the true Groovy platform, after an absence

it's been blessed by at least one (J)Groovy core developer, Andres Almiray, for use in (J)Groovy

The resulting code will be called Groovier. It'll be Apache 2.0 licensed to encourage the (J)Groovy developers to copy it for bundling with Groovy. Groovier will be to Scala and the GrAST what the (J)Groovy Language is to Java and the JVM. Groovy is getting groovier and Groovier.

Syntactic Macros in Groovy

2009-05-06T03:26:00.000-07:00

In CLisp, macros are defined by quoting symbols and lists with backquote `, and escaping from them with comma , or comma-at ,@, perhaps nestedly so. One day I realized this syntax is similar to GStrings in Groovy, which are quoted with double-quote " and escaped with dollar $, perhaps nestedly so. Lisp symbols are similar to Java interned strings. I wondered if GString syntax be used to specify syntactic macros in Groovy.

In Paul Graham's online book, On Lisp, he describes how to write a macro:
(defmacro nil! (var) `(setq ,var nil))
Using GString notation, we could define an equivalent macro in Groovy by writing:
defMacro("setq($var, null)"){"nilBang($var)"}

For Groovy to match Lisp in functionality, we would need two more functions:
'[sum, a, b]'.asFunc() == 'sum(a, b)'
'[+, a, b]'.asFunc() == '(a + b)'
and:
"[a, $b, c]".expand() == [a, 2, c]
as well as the already-provided:
"[$a, $b]".evaluate() == 3

Paul Graham writes that when using the backquote in Lisp:
`(a b c) is the same as '(a b c)
and `(a b c) is the same as (list 'a 'b 'c)
With Groovy syntactic macros:
"[a, b, c]" will be the same as '[a, b, c]'
and "[a, b, c]" the same as ["a", "b", "c"].

Using the comma with the backquote in Lisp:
`(a ,b c ,d) is the same as (list 'a b 'c d)
With Groovy macros:
"[a, $b, c, $d]" will be the same as ["a", b, "c", d]

In Lisp, if we assign some values using (setq a 1 b 2 c 3), then we'll get:

> `(a ,b c)
(A 2 C)

In Groovy, if we assign using def a=1, b=2, c=3, then we would get:
assert "[a, $b, c]".expand() == [a, 2, c]

For an example with nesting using these values:

> `(a (,b c))
(A (2 C))

In Groovy it would be:
assert "[a, [$b, c]]".expand() == [a, [2, c]]

A more complex example from On Lisp:

> `(a b ,c (',(+ a b c)) (+ a b) 'c '((,a ,b)))
(A B 3 ('6) (+ A B) 'C '((1 2)))

In Groovy it would be:

assert "[a, b, $c, ['${"[+, a, b, c]".asFunc()}'], " +
       "[+, a, b], 'c', '[[$a, $b]]']".expand() ==
  [a, b, 3, ['6'], [+, a, b], 'c', '[[1, 2]]']

Another example, in Lisp:
`(,a ,(b `,c))
and in Groovy:
"[$a, ${[b, "$c"]}]"

We'd need to introduce some additional syntax to match the comma-at notation from Lisp:

> (setq b ’(1 2 3))
(1 2 3)
> `(a ,b c)
(A (1 2 3) C)
> `(a ,@b c)
(A 1 2 3 C)

The Groovy equivalent, with additional syntax $* to show interpolate-with-spreading, would be:

def b= [1, 2, 3]
"[a, $b, c]".expand == [a, [1, 2, 3], c]
"[a, $*b, c]".expand == [a, 1, 2, 3, c]

I'm nowhere near providing this sort of functionality in Groovy/DLR, but such self-referential syntactic macros would complement the recently added AST macro system in Groovy 1.6.

The Groovy DevCons

2009-04-29T03:02:00.000-07:00

Looks like another Groovy DevCon is being held soon, perhaps to coincide with the Gr8 conference in Copenhagen, mid-May 2009, but very little info is being made public. Let's take a brief look at the history of the Groovy DevCons...

London, Nov 2004: The only info I could find about Groovy DevCon 1, then called GroovyONE, is a powerpoint presentation given by James Strachan. I remember seeing some online pics of the attendees at the time.

Paris, Nov 2005: Jeremy Rayner reports, with pics, here on the DevCon 2. Shortly afterwards, Groovy co-founder James Strachan moved on to other things.

Paris, Jan 2007: Many decisions were made, some followed, some delayed, others changed, sometimes at the last moment. Shortly after DevCon 3, John Wilson left the Groovy Developer team.

London, Oct 2007: Held concurrent with the Grails eXchange conference, where Guillaume Laforge gave this keynote, shortly after the core developers' next move to monetize their work on the language: the formation of G2One, Inc. It was just after that DevCon I heard about the discussions over changing Groovy's name. The Groovy Developers quickly plugged up that hole in their internal communications, and it seems the decision was made not to rename it.

Late last year, the Groovy/Grails controlling company, G2One Inc, was acquired by SpringSource, controller of the Spring Framework and Tomcat, in a ”stock-and-cash” deal. The SpringSource shareholders and those of G2One decided they can get acquired in a ”cash-and-stock” deal more easily if they do it together. Central to that is tightly managing all information flowing out of the Groovy development effort. So only insiders know if this coming DevCon is DevCon 5. The developers recently added Griffon to their list of officially quoted Groovy-based technologies, calling them Groovy, Grails, and Griffon, all in one breath. Now are they resisting the branding impact of my recent Groovy/DLR release by seeding the web with the ”Boo is to C# what Groovy is to Java” message? Are we going to see a Groovy-Boo marketing linkup in time for the Gr8/DevCon meeting? Where they all sing “Groovy Groovy Boo, where are you?” ?

Building Groovy/DLR
But Groovy for the DLR is NOT Groovy for .NET. It's purpose is different, simply a lexer/parser for a Groovy-like syntax, only better, and its true target is the (J)Groovy AST. I'm far more interested in programming to (J)Groovy's new ASTBuilder, as recently spec'd by Hamlet D'Arcy, than to the DLR, because (J)Groovy is the real Groovy. Groovy presently has an Antlr 2.x lexer/parser, which some say should be updated to Antlr 3.0. But others, e.g. some Grails developers, want a much faster hand-written parser instead. I'd like a slower but more extensible combinator parser. So to cater for everyone, Groovy needs clearly-specified AST node interfaces and visitation states. Perhaps the coming ASTBuilder will be like this. The Groovy AST could become the JVM's answer to .NET's DLR.

My original intention was to write the lexer/parser in C# because it had delegates (i.e. closures), then translate to Java 7.0 later. But it appears Java 7.0 is unlikely to have closures, so I could be left in limbo on the DLR. I recently got in-place comment lex-definition running, e.g. @AddComment("(?s:#{4}.*)") in the source code will cause the parser to add this comment definition to the whitespace lexer. This particular example of comment syntax is very handy, i.e. being able to add #### in the source file to comment to end-of-file, without needing to page down to add close-comment syntax at the end. I'm now starting on in-place string syntax definitions, so those who want things like multiline regexes can just add it themselves instead of waiting years for the (J)Groovy developers to do so.

Underwriting (J)Groovy
Besides building Groovy/DLR, another involvement I have with the Groovy language is underwriting it: Should the Groovy Language ever have its development abandoned or its name changed, I, as Groovy's underwriter, will pick up from where they left off. When I worked in IT, I found being the ”Backup Programmer” for a project to be quite difficult because I never knew how much time and effort to put into following something I would probably never work on. Many a business manager, having little understanding about insuring against risks, has slashed programmer jobs on a computer system to cut costs, making others bear the true costs later on. And so it is with underwriting the Groovy Language. Unlike that of the Groovy Developers, who manage Groovy's upside, my performance in covering Groovy's downside can't be measured unless I actually need to do something.

I've tried to participate in Groovy's success in other ways, such as writing Groovy documentation for Java newbies. I was being encouraged publicly, but got a different message from somewhere through backchannels, telling me to “stop messing with the brand”. I didn't know for sure where the antagonism was coming from, nor can I prove it, but had suspicions when I read this excellent analysis on brand power in open source software. It seems a name is not just a game. Because of this, I'm quite hesitant to join the Codehaus Groovy effort, prefering to work on the sidelines both underwriting Groovy, and increasing Groovy programmers' lexical and syntactic choices. The Groovy Language will give more choices to developers.

Interactional function of English and Groovy

2009-04-23T07:39:00.000-07:00

Michael A.K. Halliday writes in his 1970 paper Language Structure and Language Function that we should analyze language in terms of its use, considering both its structure and function in so doing. He's found the vast numbers of options embodied in it combine into three relatively independent components, and they each correspond to a certain basic function of language: representational (a.k.a. ideational), interactional (a.k.a. interpersonal), and textual. Within each component, the networks of options are closely interconnected, while between components, the connections are few.

For natural language, the representational component represents our experience of the outside world, and of our consciousness within us. The representational similarities between natural and computer languages are most easily noticed:

Mary.have(@Little lamb)
lamb.fleece.Color = Color.SNOW_WHITE
synchronized{ place-> Mary.go(place); lamb.go(place) }

Computer languages' increasing use of abstraction over the years was no doubt based on the representational component of natural languages, giving rise to the functional and object-oriented paradigms. The ideas represented in computer language must be more precise than those in natural language.

Interactional component of English
The interactional component of language involves its producer and receiver/s, and the relationship between them. For natural language, there's one or more human receivers, and for computer language, one or more electronic producers and/or receivers as well as the human one/s.

In English, the interactional component accounts for:

many adverbs of opinion, e.g. “That's an incredibly interesting piece of code!”

interjections within a clause, e.g. I'm hoping to, er, well, go back sometime, or even in the middle of words, e.g. abso-bloomin'-lutely

expressions of politeness we prepend to English sentences, e.g. “Are you able to...” in front of “Tell me the time”

the hundreds of different attitudinal intonations we overlay onto our speech, e.g. ”Dunno!” (can you native English speakers hear that intonation?)

the mood, whether indicative e.g. “He's gone.”, interrogative e.g. ”Is she there?”, imperative e.g. ”Go now!”, or exclamative e.g. ”How clever!”

the modal structure of English grammar, i.e. verbal phrases have certainty e.g. “I might see him”, ability e.g. ”I can see her”, allowability e.g. ”Can he do that?”, polarity e.g. ”They didn't know”, and/or tense e.g. ”We did make it”

Natural language offers many choices regarding how closely to intertwine the interactional component with the representational.
An example... for closely intertwined reported speech:

She said that she had already visited her brother, that the day before she'd been with her teacher, and that at that moment she was shopping with her friend.

and using quoted speech to reduce the tangling between interactional and representational components: She said "I've already visited my brother, yesterday I was with my teacher, and right now I'm shopping with my friend."
Another example of keeping these two components disjoint: I'm going to tell the following story exactly as she told it, the way she said it, not how I'd say it...

The original human languages long ago, just like chimpanzee language today, was perhaps mainly interactional, with the representional component slowly added on afterwards.

Interactional component of computer languages
For computer languages, the interactional component determines how people interact with the program, and how other programs interact with it. Like natural languages, the interactional component came first, and representational abstractions added on later. Many have tried to create a representational-only computer language, perhaps the most successful is Haskell. But the Haskell language creators went to great trouble to tack on the minimumly required interactional component, that of Input/Output. They introduced monads to add the I/O capability onto the “purer” underlying functional-paradigm function. Perhaps some functional-paradigm language creators don't appreciate the centrality of the interactional component in language.

Siobhan Clarke et al, writes about the tyranny of the dominant decomposition:

Current object-oriented design methods suffer from the “tyranny of the dominant decomposition” problem, where the dominant decomposition dimension is by object. As a result, designs are caught in the middle of a significant structural misalignment between requirements and code. The units of abstraction and decomposition of object-oriented designs align well with object-oriented code, as both are written in the object-oriented paradigm, and focus on interfaces, classes and methods. However, requirements specifications tend to relate to major concepts in the end user domain, or capabilities like synchronisation, persistence, and failure handling, etc., all of which are unsuited to the object-oriented paradigm.

The object-paradigm is a representational one. The other user-domain capabilities are interactional ones, either human-to-computer or computer-to-computer. Some examples:

I/O actions, i.e. between computer and human/s

logging, i.e. between processor and recording medium

persistence, database access, i.e. between computer and storage unit/s

security, i.e. between computer and certain humans only

execution performance, i.e. how to maximize use of computing resources

entry point to program, i.e. between procesor and external scheduler

concurrency, synchronization, i.e. between two processors in one computer

distribution, i.e. between two geographically separated computers

exceptions, failure handling, i.e. between results of different human-expected certainties

testing, i.e. interaction between two different external humans

These capabilities are often interwoven into the programming code, just as mood and modality are overlaid onto all the finite verbal phrases in English. And just as in English, where various interactional functions can be disentangled from the representation functions, e.g. quoted speech above, so also in computer languages, such user-domain capabilities can be extracted as system-wide aspects in aspect-oriented programming.

AspectJ is a well-known attempt to let each aspect use the same syntax, that of the base language. But the idea of limited AOP is much older, often different syntaxes are used for each different user-domain capability.

Finally...
I've already blogged about some aspects of the textual component of English and Groovy. Whereas the other two components of language exist for reasons independent of the medium itself, the textual component comes into being because the other two components exist, and refers both to those other two components and to itself self-referentially. The textual component ensures every degree of freedom available in the medium itself is utilized.

In computer languages, the textual component if often called ”syntactic sugar”. Often computer language designers scorn the use of lots of syntactic sugar, but natural language designers, i.e. the speakers of natural languages, use all the syntactic sugar available in the communication medium. Programming languages designers should do the same. In the DLR-targetted Groovy I'm working on, I'm focusing on this aspect of the Groovy Language.

Groovy for the DLR

2009-04-10T22:39:00.000-07:00

The source code for beta-3 of Groovy/DLR 1.0 is out. Groovy for the Microsoft DLR is an Apache-2-licenced combinator parsing library that lexes and parses (J)Groovy-lookalike syntax to build DLR nodes.

What can execute...
I've now successfully plugged in all of the node builds from the ToyScript sample language in DLR version 0.9. Here's an example of syntax beta-03 can parse and execute...

//various atomic values...
print 444;
print - 34;
print('abc') // <--- implied semis to end statements
print 7.515
print(true);
print(3+7*
  5); //multi-line statement
print(3+7*
  5) //multi-line stmt with implied semis
a = 2*3000;
print a;
print 12 + a * 2;
pause

//we can access .NET classes...
import System
zz= new System.Collections.ArrayList();
zz.Add(7)
zz.Add('abc')
zz.Add(6.66)
print zz.Count
print zz[0]
print zz[1]
print zz[2]
print zz[1] == 'abc'
print zz[1]
pause

try{ //while and if statements...
  i = 0;
  while(i < 5){
    i=i+ 1;
    print i
  }
  if(i<1){print '...first...';}else{print '...second...';};
  pause;
};

def add(a,b) { //function defn and invocation...
  j = a + b;
  return j
}
print add(1, 2)
print add('Hello, ', 'Groovy/DLR.')
pause

//'parse' is a temporary command to parse, but not run, another file...
parse 'test/test01.gvy';
parse 'test/test02.gvy';
parse 'test/test03.gvy'
parse 'test/test04.gvy'
pause

Groovy/DLR can parse more syntax than it can execute, including GStrings, closures, and classes, which are tested in extra parse-only files.

Goals (reposted)
My key goal when building the Groovy/DLR Language syntax has been to make it totally configurable by the programmer. That's why I've used combinator parsers, instead of bottom-up parsing. Programmers can easily read and change what combinator parsers do. They can swap a 3-character operator symbol e.g. <=> with a single-character Unicode one. They can put in a lookup table to map method names in French or Chinese to ones in English. They can rewrite the entire grammar, based on an in-house corporate policy, but keep the same AST tree semantics. They can embed other little languages in the grammar, without quoting. They can change the tokenizer rules to embed multiline strings with any escapes they want.

Some salient features of these combinator parsers:

Lexing and parsing GStrings, which can be embedded to any number of levels, was an early challenge. When lexing, we return a list of tokens, some of which could be fully parsed GStrings. None of the parsing logic is in the lexer, instead, the lexer calls the parser repeatedly to find the end of GString enclosures.

After building the lexer and then the parser, we query the parser to get a list of operator tokens, then pass this to the lexer before running it. That way, we specify tokens only once, in the parser, not in the lexer. The DRY principle at work.

The lexing and parsing libraries are separate, fine-tuned for their specific tasks. Not worrying about how a feature required for the parsing can work for the lexing, and vice versa, caused the lexer and parser to evolve into two hugely different beasts.

When parsing statements and expressions, we don't litter the code with newline checks, like (J)Groovy does, instead this is separated out into a separate context-sensitive pass using monadic parsers (i.e. bind and return). That way, programmers can add their own statement or expression parsers with the minimum of code, without worrying about the separate concern of how the statement fits into the overall syntactic structure.

The original source code can be completely rebuilt from the AST nodes. All whitespace is stored on the tokens.

Unlike my previous code and Groovy's Antlr-based parser and AST generation code, this C#-based code uses objects and visitation a lot. Example: the IfStatementToken contains a method returning the combinator parser, a method for generating that portion of the original source, a method for generating source code in a (J)Groovy-compatible normal form, a method for generating the DLR nodes, etc.

Future plans...
I now working on getting Python-style indenting blocks working, using syntax {:, so eventually the developer can mix close-curly-delimited and indented blocks in the same code. When this is working, I'll separate out the code into a separate generic combinator parser, then reapply it using token (: to argument/parameter lists, and token [: to list/map constructors, so developers can mix close-delimited and indented structures of any kind in their code.

I'm hoping to eventually enable a syntactic macro style whereby we can define syntax, then use it in the same source code, e.g.

public class WhileStmtToken: StatementToken {:
  NameToken WhileTag
  ExpressionToken Expr
  BlockToken Block

  static Parser BuildParser( //how to parse the syntax...
    Parser parsery, Parser exprParser, Parser blockParser) {:
      parsery.Name("while")
      .Seq(exprParser){:
        whileTag, expr->
        new WhileStmtToken{:
          Expr = expr as ExpressionToken
          WhileTag = whileTag as NameToken
      .Seq(blockParser){:
        token, block->
        (token as WhileStmtToken).Block = block as BlockToken
        token

  Expression Generate(Generator gen) {: //how to generate the DLR node...
    Utils.While(:
      gen.ConvertTo(typeof(bool), Expr.Generate(gen)),
      Block.Generate(gen),
      null, null, null, ss.End, ss

  string ToString(int indent) {:
    //...etc...
  string NormalForm(int indent) {:
    //...etc...
  string SourceCode() {:
    //...etc...

useSyntax(WhileStmtToken){: //use class above when parsing code below
  while(condition){:
    codeToRun()

I'm hesitant to put in any more DLR nodes until DLR version 1.0 is out, because things might change. Perhaps the ToyScript sample language shipped with DLR 1.0 will have more structures implemented so I can copy them. I'm focusing on the combinator parsers because those are what I want copied by someone and put into (J)Groovy. One big hope I have for the Groovy/DLR language is it'll inspire the (J)Groovy developers at Codehaus to also enable a flexible syntactic skin in (J)Groovy for the JVM, the Groovy Language's first target platform. (J)Groovy originally used a handwritten bottom-up parser: no doubt it parsed quicker, but it couldn't easily be expanded with new syntax. After 10 betas, (J)Groovy switched to an Antlr 2.x-based parser, initially allowing more syntax to be added, but now has become so complex the (J)Groovy Developers don't want to add much new syntax, nor even allow developers to customize the parsing themselves. A true DSL needs a lot more flexibility than parenthesis-less method calls and annotation-driven AST transformations. The (J)Groovy AST could become the JVM's answer to the Microsoft DLR.

Groovy Combinator Parsing

2009-03-16T04:48:00.000-07:00

About 18 months ago, I began writing an alternative lexer and parser for the Groovy AST. My original plan was to write different layers of code:

an ASTBuilder using Groovy's builder facility
a lexer and parser using Ben Yu's JParsec 1.2 combinator parsing library, calling the ASTBuilder in the closures
an AST decompiler to Groovy source

The ASTBuilder was difficult to build so it would be callable from closures embedded within JParsec. At the time I concluded the builder pattern that Groovy builders embody was unsuitable for building AST's, and that the interpreter pattern would be better. (Perhaps that's why programming languages became the preferred method to interface to AST's way back :-) So I switched to building the AST directly in the JParsec closures. I called all this Vy Language 1.0.

However, I eventually abandoned development. The reasons:

although JParsec 1.2 was well-behaved, without bugs, I found it brittle when used for a large grammar of my own design
my development methodology was similar to what I'd done in previous paid employment, i.e. get it working ASAP, meet the deadline, and don't worry about bugs, quality, and documentation till later. The resulting code worked (tho with bugs), but was messy.
I tried to reproduce the entire Groovy 1.0 grammar
I got the GString lexing/parsing working (incl nested GStrings) using monadic parsers, but not the implied semicolons
the layering design was wrong
the closures I gave JParsec expanded out to 2kb each, all 200-odd of them
the Groovy AST was a moving target, with no spec, and the Groovy Developers weren't adding the setters I needed to the AST nodes

Second try...

I again tried writing a lexer/parser for the Groovy AST, this time using my own combinator parsers, which I based on Ken Barclay's ones written in Groovy. I called it Vy Language 1.1. Ken later announced GParsec, where he would use his ones to reproduce the Groovy grammar, though 1 yr later, no code's appeared. Perhaps he had the same problems I had:

the code ran slooooowly, all of it, not just the exponential edge-cases
although I implemented a lot less functionality than for my earlier version (v 1.0), the code produced over 300 closures, each at 2kb

Third try...

After a break, I was still excited about building a computer language grammar with all the cool ideas I've had while studying natural language theory. So late last year, I tried again, but didn't want to repeat my previous mistakes. So:

I needed a fast language with closures. Java was fast but doesn't have closures; Groovy has closures but runs slowly. So I started playing with C#.
Instead of whacking out as much code as possible, as soon as possible, I've decided to enjoy the process and refactor often.
The Microsoft DLR version 0.9 is available. I've recently fitted some of the nodes from the ToyScript sample language into my grammar.

I've put the source code online, calling it Groovy for the DLR. It's really just a combinator parsing library, with some Groovy-like grammar, and some DLR nodes created. I didn't start out intending to build Groovy for the DLR, but it's natural evolution is exactly that.

I'm now deciding whether to continue down that lane, or to switch back to the JVM. Because the code is closure-heavy (i.e. C# delegates), it'd be difficult but doable to port it to Java (JParsec manages to implement such stuff in Java without closures). If Java 7 has closures built in, that would help. And of course I would need some target platform.

Guillaume Laforge mentions AST builders in the most recent Groovy roadmap, but does that mean an openly-specified AST interface available for everyone to use, or an elusively ever-changing implementation that only works with the insiders' bundled builders? There's an opportunity here for the Groovy AST to become the JVM's answer to the Microsoft DLR. Will the Groovy Developers grasp the opportunity?

Groovy 1.6 Released

2009-02-23T23:39:00.000-08:00

Groovy 1.6 has recently been released, the first update to the Groovy Language since version 1.5 over a year ago. Version 1.6 brings Python-style tuples to Groovy, allowing multi-value assignments. This was the last unfulfilled item from James Strachan's original 29 August 2003 manifesto for Groovy.

But many items remain unfulfilled from Groovy's submission as JSR 241 on 16 March 2004 and subsequent approval 2 weeks later. James expanded on his vision at that time. Let's look at some of his points...

(1) James wrote: “

One area we've not yet looked at is merging a Java and Groovy compiler together so Groovy and Java code can be compiled together in the same compilation unit.

” The Groovy developers added this capability in their implementation of Groovy 1.5 a year ago.

(2) He also wrote: “

The JSR allows people to [implement Groovy] if they wish - whether its a complete rewrite, replacing a part of it or just some tinkering, and provides a TCK to know if they correctly implement the JSR. Even just embedding the RI inside a container can sometimes affect things so having a JSR & TCK helps even if we're sharing the same codebase but just configuring & deploying it in different ways.

.” A spec and test kit encourages other developers to come along and contribute to the language correctly, improving it or the parts that need it. This was the vision in place when I first started tinkering with Groovy.

(3) Also: “

Just to be clear, its the expert group's responsibility to make a great spec and a reference implementation and TCK. It's up to others to create different implementations if they want to

.” (his emphasis) Not having a JSR provides only one avenue for contributing to the Groovy Language, that of joining the development team, submitting to the commercial interests and internal politics of Codehaus and SpringSource and whoever else might come along, something very few developers want to do. But having a completed JSR provides many avenues to contribute to Groovy. A spec and test kit are like the market rules for a bazaar, but the present Groovy development is still a centrally-planned cathedral. The only really committed developers are those with shares in the business, and even then, when they're not working on Spring or Grails.

Having tight control over the initial stages of an open-source software package keeps development focused. A year after development began, Groovy creator James Strachan thought the time was right for creating a spec and test kit. After some initial activity, work on the spec halted, the reason given was “possible copyright violations with Sun's Java language spec”. The Groovy developers since then have focused on creating the fastest possible JVM meta-object engine, and on melding their implementation of Groovy closely into Grails, both good activities but not an end in themselves.

(4) Finally, James also added: “

I'm hoping Groovy goes along the same way - that one day maybe Jikes / gcj implement a Groovy compiler or maybe IDEs (say in Eclipse / IDEA / NetBeans / Workshop) do their own Groovy compiler, reusing their internal Java compilers - maybe reusing the Groovy AST compiler, just replacing the bytecode generation with something else, like Java AST generation, Java code generation or whatever.

” James Strachan envisioned Groovy as a loose collection of language technologies, just as the Spring technologies are. A Groovy spec, therefore, needs to define not only the language syntax, but also the meta-object interface, the default Groovy method interface, the AST interface, with clearly-defined visitation states, and so forth.

The Groovy developers, to their credit, have now completed all the items in James' original manifesto, as well as adding a joint Groovy/Java compiler. But the JSR hasn't moved an inch in the past 5 years. Let's hope Groovy 1.7 (or whatever it's called) can address this issue.

The Rise of Unicode

2009-02-12T20:31:00.000-08:00

The next version of Unicode is v.5.2, the latest of a unified character set now with over 100,000 current tokens. One notable addition to v.5.2 will be the Egyptian hieroglyphs, the earliest known system of human writing. Perhaps they will mark Unicode's coming of age, it being another huge step in representing language with graphical symbols. Let's look at a consolidated short history of writing systems, courtesy of various Wikipedia pages, to see Unicode's rise in perspective...

Hieroglyphs
Egyptian hieroglyphs were invented around 4000-3000 BC. The earliest type of hieroglyph was the logogram, where a common noun (such as sun or mountain) is represented by a simple picture. These existing hieroglyphs were then used as phonograms, to denote more abstract ideas with the same sound. Later, these were modified by extra trailing hieroglyphs, called semagrams, to clarify their meaning in context. About 5000 Egyptian hieroglyphs existed by Roman times. When papyrus replaced stone tablets, the hieroglyphs were simplified to accommodate the new medium, sometimes losing their resemblance to the original picture.

The idea of such hieroglyphic writing quickly spread to Sumeria, and eventually to ancient China. The ancient Egyptian and Sumerian hieroglyphs are no longer used, but modern Chinese characters are descended directly from the ancient Chinese ones. Because Chinese characters spread to Japan and ancient Korea, they're now called CJK characters. By looking at such CJK characters, we can get some idea of how Egyptian hieroglyphs worked. Many CJK characters were originally pictures, such as 日 for sun, 月 for moon, 田 for field, 水 for water, 山 for mountain, 女 for woman, and 子 for child. Some pictures have meanings composed of other meanings, such as 女 (woman) and 子 (child) combining into 好, meaning good. About 80% of Chinese characters are phonetic, consisting of two parts, one semantic, the other primarily phonetic, e.g. 土 sounds like tu, and 口 means mouth, so 吐 also sounds like tu, and means to spit (with the mouth). The phonetic part of many phonetic characters often also provides secondary semantics to the character, e.g. the phonetic 土 (in 吐) means ground, where the spit ends up.

Alphabets
Eventually in Egypt, a set of 24 hieroglyphs called uniliterals evolved, each denoting one consonant sound in ancient Egyptian speech, though they were probably only used for transliterating foreign names. This idea was copied by the Phoenicians by 1200BC, and their symbols spread around the Middle East into various other languages' writing systems, having a major social effect. It's the base of almost all alphabets used in the world today, except CJK characters. These Phoenician symbols for consonants were copied by the ancient Hebrews and for Arabic, but when the Greeks copied them, they adapted the symbols of unused consonants for vowel sounds, becoming the first writing system to represent both consonants and vowels.

Over time, cursive versions of letters evolved for the Latin, Greek, and Cyrillic alphabets so people could write them easily on paper. They used either the block or the cursive letters, but not both, in one document. The Carolingian minuscule became the standard cursive script for the Latin alphabet in Europe from 800AD. Soon after, it became common to mix block (uppercase) and cursive (lowercase) letters in the same document. The most common system was to capitalize the first letter of each sentence and of each noun. Chinese characters have only one case, but that may change soon. Simplified characters were invented in 1950's mainland China, replacing the more complex characters still used in Hong Kong, Taiwan, and western countries. Nowadays in mainland China though, both complex and simplified Chinese are sometimes used in the same document, the complex ones for more formal parts of the document. Perhaps one day complex characters will sometimes mix with simplified ones in the same sentence, turning Chinese into another two-case writing system.

Punctuation
Punctuation was popularized in Europe around the same time as cursive letters. Punctuation is chiefly used to indicate stress, pause, and tone when reading aloud. Underlining is a common way of indicating stress. In English, the comma, semicolon, colon, and period (,;:.) indicated pauses of varying degrees, though nowadays, only comma and period is used much in writing. The question mark (?) replaces the period to indicate a question, of either rising or falling tone; the exclamation mark (!) indicates a sharp falling tone.

The idea of separating words with a special mark also began with the Phoenicians. Irish monks began using spaces in 600-700AD, and this quickly spread throughout Europe. Nowadays, the CJK languages are the only major languages not using some form of word separation. Until recently, the Chinese didn't recognize the concept of word in their language, only of (syllabic) character.

The bracketing function of spoken English is usually performed by saying something at a higher or lower pitch, between two pauses. At first, only the pauses were shown in writing, perhaps by pairs of commas. Hyphens might replace spaces between words to show which ones are grouped together. Eventually, explicit bracketing symbols were introduced at the beginning and end of the bracketed text. Sometimes the same symbol was used to show both the beginning and the end, such as pairs of dashes to indicate appositives, and pairs of quotes, either single or double, to indicate speech. Sometimes different paired symbols were used, such as parentheses ( and ). In the 1700's, Spanish introduced inverted ? and ! at the beginning of clauses, in addition to the right-way-up ones at the end, to bracket questions and exclamations. Paragraphs are another bracketing technique, being indicated by indentation.

Printing
Around 1050, movable-type printing was invented in China. Instead of carving an entire page on one block as in block printing, each character was on a separate tiny block. These were fastened together into a plate to reflect a page of a book, and after printing, the plate was broken up and the characters reused. But because thousands of characters needed to be stored and manipulated, making movable-type printing difficult, it never replaced block printing in China. But less than a hundred letters and symbols need to be manipulated for European alphabets, much easier. So when movable-type printing reached Europe, the printing revolution began.

With printing a new type of language matured, one that couldn't be spoken very well, only written: the language of mathematics. Mathematics, unlike natural languages, needs to be precisely represented. Natural languages are very expressive, but can also be quite vague. Numbers were represented by many symbols in ancient Egypt and Sumeria, and had reduced to a mere 10 by the Renaissance. But from then on, mathematics started requiring many more symbols than merely two cases of 26 letters, 10 digits, and some operators. Many symbols were imported from other alphabets, different fonts introduced for Latin letters, and many more symbols invented to accommodate the requirements of writing mathematics. Mathematical symbols are now almost standardized throughout the world. Many other symbol systems, such as those for chemistry, music, and architecture, also require precise representation. Existing writing systems changed to utilize the extra expressiveness that came with movable-type printing. Underlining in handwriting was supplemented with bolding and italics. Parentheses were supplemented with brackets [] and curlies {}.

Fifty years ago, yet another type of language arose, for specifying algorithms: computer languages. The first computer languages were easy to parse, requiring little backtracking, but the most popular syntax, that of C and its descendants, requires more complex logic and greater resources to parse. Most programming languages used a small repetoire of letters, digits, punctuation, and symbols, being limited by the keyboard. Other languages, most notably APL, attempted to use many more, but this never became popular. Unlike mathematics, computer languages relied on parsing syntax, rather than a large variety of tokens, to represent algorithms, being limited by the keyboard. Computer programs generally copied natural language writing systems, using letters, numbers, bracketing, separators, punctuation, and symbols in similar ways. One notable innovation of computer languages, though, is camel case, popularized for names in C-like language syntaxes.

The natural language that spread around the world in modern times, English, doesn't use a strict pronunciation-spelling correspondence, perhaps one of the many reasons it spread so rapidly. English writing therefore caters for people who speak English with widely differing vowel sounds and stress, pause, and tone patterns. In this way, English words are a little like Chinese ideographs. As Asian economies developed, techniques for quickly entering large-character-set natural languages were invented, known as IME's (input method editors). But these Asian countries still use English for computer programming.

Unification
Around 1990 Unicode was born, unifying the character sets of the world. Initially, there was only room for about 60,000 tokens in Unicode, so the CJK characters of China, Japan, and Korea were unified to save space. Unicode is also bidirectional, catering to Arabic and Hebrew. Topdown languages such as Mongolian and traditional Chinese script can be simulated with left-to-right or right-to-left directioning by using a special sideways font. However, Unicode didn't become very popular until its UTF-8 encoding was invented 10 years ago, allowing backwards compatibility with ASCII. Another benefit of UTF-8 is there's now room for about one million characters in the Unicode character set, allowing less commonly used scripts such as Egyptian hieroglyphs to be encoded.

Many programming languages have recently adopted different policies for using Unicode tokens in names and operators. The display tokens in Unicode are divided into various categories and subcategories, mirroring their use in natural language writing systems. Examples of such subcategories are: uppercase letters (Lu), lowercase ones (Ll), digits (Nd), non-spacing combining marks, e.g. accents (Mn), spacing combining marks, e.g. Eastern vowel signs (Mc), enclosing marks (Me), invisible separators that take up space (Zs), math symbols (Sm), currency symbols (Sc), start bracketing punctuation (Ps), end bracketing (Pe), initial quote (Pi), final quote (Pf), and connector punctuation, e.g. underscore (Pc).

For it to become popular to use a greater variety of Unicode tokens in computer programs, there must be a commonly available IME for their entry with keyboards. Sun's Fortress provides keystroke sequences for entering mathematical symbols in programs, but leaves it vague whether the Unicode tokens or the ASCII keys used to enter them are the true tokens in the program text. And of course there must be a commonly available font representing every token. Perhaps because of the large number of CJK characters, and the recent technological development of mainland China, a large number of programmers may one day suddenly begin using them in computer programming to make their programs terser.

Conclusion
Language representation using graphical symbols has taken many huge leaps in history: Egyptian hieroglyphs to represent speech around 4000 years ago, an alphabet to represent consonant and vowel sounds by the Phoenicians and Greeks around 2500 years ago, movable-type printing in Europe around 500 years ago, and unifying the world's alphabets and symbols into Unicode a mere 20 years ago. And who knows what the full impact of this latest huge leap will be?

The Thematic Structure of English and Groovy

2008-12-06T02:15:00.000-08:00

After working as a programmer for many years, I tossed it in to teach English in China. I spent a few years reading the many books on language and linguistics in the bookshops up here, before returning to programming as a hobby. I then started to see many similarities between natural and computer languages, which I'm intermittently blogging about. Here's today's installment...

Introduction
Of the books on language I've read, M.A.K. Halliday's ideas make a lot of sense. He suggests we should analyse language in terms of what it's used for, rather than its inherent structure. From this basis, he's isolated three basic functions of natural language, and their corresponding structural subsystems: the ideational, the interpersonal, and the textual.

The ideational function is a representation of experience of the outside world, and of our consciousness within us. It has two main components: the experiential and the logical. The experiential component embodies single processes, with their participants and circumstances, in a transitivity structure. For example, “At quarter past four, the train from Newcastle will arrive at the central station.” has a transitive structure with process to arrive, participants train from Newcastle and central station, and circumstance quarter past four. The primary participant is called the actor, here, the train from Newcastle. Computer languages have a structure paralleling the transitivity structure of natural languages, e.g. train.arrive(station, injectedCircumstance) for object-oriented languages. The logical component of ideational function concerns links between the experiential components, attained with English words such as and, which, and while. These have obvious parallels in programming languages.

The interpersonal function involves the producer and receiver/s of language, and the relationship between them. This function accounts for the hundreds of different attitudinal intonations we overlay onto our speech, interjections, expressions of politeness we prepend to English sentences, e.g. “Are you able to...”, many adverbs of opinion, the mood (whether indicative, interrogative, imperative, or exclamative), and the modal structure of English grammar. The mood structure causes verbal phrases to have certainty, ability, allowability, polarity, and/or tense prepended in English, and can be repeated in the question tag, e.g. isn't he?, can't we?, should they?. The interpersonal function gives the grammatical subject-and-predicate structure to English. In programming languages, the interpersonal function determines how people interact with the program, and how other programs interact with it. The interpersonal functions are what would normally be extracted into aspects in aspect-oriented programming. They generally disrupt the “purer” transitivity structure of the languages.

The textual function brings context to the language through different subsystems. The informational subsystem divides the speech or text into tone units using pauses, then gives stress/es to that part of the unit that is new information. The cohesive subsystem enables links between sentences, using conjunctions and pronouns, substitution, ellipsis, etc. The thematic subsystem makes it easy for receivers to follow the flow of thought. Comparing this structure of the English and Groovy languages is the topic of today's blog post...

Thematic structure of English
Theme in English is overlaid onto the clause, a product of the transitive and modal structures. The theme is the first position in the clause. English grammar allows any lexical item from the clause to be placed in first position. (In fact, English allows the lexical items to be arranged in any order to enable them to be broken up in any combination into tone units.) Some examples, using lexical items to give, Alan, me, and the book, with theme bolded:
  Alan gave me that book in London.
  Alan gave that book to me in London. (putting indirect object into prepositional phrase)
  To me Alan gave that book in London. (fronting indirect object)
  I am who Alan gave that book to in London. (fronting indirect object, with extra emphasis)
  To me that book was given in London. (using passive voice to front indirect object)
  That book was given in London. (using passive voice to omit indirect object)
  That book Alan gave me in London. (fronting direct object as topic)
  That book is the one Alan gave me in London. (fronting direct object in more formal register)
  In London, Alan gave me that book. (fronting adverbial, into separate tone unit)
  London is where Alan gave me that book. (fronting adverbial in the same tone unit)
  There is a book given by Alan to me in London. (null topic)

Although not common, English also allows the verb to be put in first position as theme:
  Give the book Alan did to me in London.
  Give me the book did Alan in London.
  Give did Alan of the book to me in London.
  Given was the book by Alan to me in London.

First position is merely the way English indicates what the theme is, not the definition of it. Japanese indicates the theme in the grammatical structure (with the inflection はwa), while Chinese (I think) uses a combination of first position and grammatical structure (prepending with 是shi).

Thematic structure of Groovy
One way of indicating theme could be to bold it, assuming the text editor had rich text capabilities. This would similar to Japanese. For example, for thematic variable a
  def b = a * 2; def c = a / 2;.
Another way is to use first position, how English indicates it. This would be an Anglo-centric thematic structure to programming languages, which generally already have an Anglo-centric naming system. Perhaps the best way is a combination of both front position and bolding.

Let's look at how Groovy could enable front-position thematic structure. We'll start with something simple, the lowest precedence operator: a = b. If we want to front the b, we can't. We would need some syntax like =:, the reverse of Algol's :=
  b =: a

We'd need to provide the same facility for the other precedence operators at the same level += -= *= /= %= <<= >>= >>>= &= ^= |=. Therefore, we'd have operators =+: =-: =*: =/: =%: =<<: =>>: =>>>: =&: =^: =|:.

At the next higher precedence level are the conditional and Elvis operators. Many programming languages, such as Perl and Ruby, enable unless as statement suffix, allowing the action to be fronted as the theme. Groovy users frequently request this feature of Groovy on the user mailing list. An unless keyword would be useful, but we could also make the ? : and ?: operators multi-theme-enabling by reversing them, i.e. : ? and :?, with opposite (leftwards) associativity. The right-associative ones would have higher precedence over these new ones, so, for example:
  a ? b : c ? d : e would associate like a ? b : (c ? d : e)
  a : b ? c : d ? e would associate like (a : b ? c) : d ? e
  a : b : c ? d ? e would associate like a : (b : c ? d) ? e
  and a ? b ? c : d : e would associate like a ? (b ? c : d) : e

On a similar note: Groovy dropped the do while statement because of parser ambiguities. It should be renamed do until to overcome the ambiguities.

Next up the precedence hierarchy, we need shortcut boolean operators ||: and &&:, which evaluate, associate, and shortcut rightwards. Most of the next few operators up the hierarchy | ^ & == != <=> < <= > >= + * don't need reverse versions, but these do: =~ ==~ << >> >>> - / % **. It's good Groovy supplies the ..< operator so we can emphasize an endpoint in a range without actually processing it. We'll also provide the >.. and >..< operators.

Just as in English we have the choice of saying the king's men or the men of the king, depending on what we want to make thematic, we should have that choice in Groovy too.
We can easily encode reverse-associating versions of *. ?. .& .@ *.@ as .* .? &. @. @.*. To encode the standard path operator ., we could use .:.

A positive by-product of having these reverse-associative versions of the Groovy operators is they'll work nicely with names in right-directional alphabets, such as Arabic and Hebrew, when we eventually enable that.

When defining methods in Groovy, we should have the choice to put return values and modifiers after the method name and parameters, like in Pascal. This would cater speakers of Romance languages, e.g. French, who generally put the adjectives after the nouns.

Conclusion
Groovy, like most programming languages, doesn't enable programmers to supply their own thematic structure to code, only the transitive structure. When used well, thematic structure in code enables someone later on to more easily read and understand the program. Perl was a brave attempt at providing “more than one way to do things”, but most programming languages haven't learnt from it. I'm working on a preprocessor for the Groovy Language, experimenting with some of these ideas. If it looks practical, I'll release it one day, as GroovyScript. It will make Perl code look like utter verbosity.

GrerlVy Symbology

2008-11-29T23:12:00.000-08:00

In my previous post, I suggested we should reserve alphanumeric tokens (Unicode categories L, N, M) for lexical items (e.g. names and numbers) in a programming language, and the other tokens (categories S, P, Z) for grammatical items. That way, people can use any alphanumeric tokens for user-defined names. Programs are also terser because programmers don't need to put whitespaces between name and symbol, or (usually) between two symbols.

The grammatical tokens should have less emphasis than the lexical ones. When a programming language uses names for grammatical items, it's not following the way of natural languages, which can confuse those new to programming. Just as unusual, those same grammatical words are bolded in many IDE's, not the lexical ones. In Fortran these grammatical words were even capitalized, while user-defined names weren't. Programming languages should follow natural languages, or else programmers can't analyze and model the problem space in programs as easily.

Let's look at some of the culprits...

Keywords
Perhaps Smalltalk has the least number of keywords:
true false nil self super thisContext

The first language I ever programmed in was a dialect of Algol60, the forerunner of C. We can see there the origins of today's well-known keywords:

true false boolean integer real string array procedure own value label comment while do for step until if then else switch goto begin end

C had 32 keywords:

auto break case char const continue default do double else enum extern float for goto if int long register return short signed sizeof static struct switch typedef union unsigned void volatile while

C++, besides using those in C, added these many more:

asm bool catch class const_cast delete dynamic_cast explicit export false friend inline mutable namespace new operator private protected public reinterpret_cast static_cast template this throw true try typeid typename using virtual wchar_t

C# has 79 keywords, removing a few of C++'s, but adding all these:

abstract as base byte checked decimal delegate event finally fixed foreach get implicit in interface internal is lock null object out override params readonly ref sbyte sealed set stackalloc string typeof uint ulong unchecked unsafe

though 2 of them, get and set, are only keywords in the right context.

Java has 53 keywords:

abstract assert boolean break byte case catch char class continue default do double else extends final finally float for if implements import instanceof int interface long native new package private protected public return short static super switch synchronized this throw throws transient try void volatile while true false null

Groovy adds in as it and def to these.

It's very difficult to extend a language without adding keywords. AspectJ adds many to Java:

aspect aspectOf() issingleton() perthis() pertarget() percflow() percflowbelow() privileged declare precedence parents error warning soft thisJoinPoint thisEnclosingJoinPointStaticPart pointcut before around after returning throwing execution call get set target args initialization preinitialization staticinitialization handler adviceexecution within withincode cflow cflowbelow proceed()

though many are sensitive to the context.

Python has a more frugal attitude to keywords:

False None True class continue def and as assert break finally for from del elif else except is lambda nonlocal global if import in return try while not or pass raise with yield

Likewise, Ruby:

BEGIN END alias and begin break case class def defined do else elsif end ensure false for if in module next nil not or redo rescue retry return self super then true undef unless until when while yield

PHP is far more wasteful of the naming space:

and or xor __FILE__ exception __LINE__ array() as break case class const continue declare default die() do echo() else elseif empty() enddeclare endfor endforeach endif endswitch endwhile eval() exit() extends for foreach function global if include() include_once() isset() list() new print() require() require_once() return() static switch unset() use var while __FUNCTION__ __CLASS__ __METHOD__ final php_user_filter interface implements instanceof public private protected abstract clone try catch throw cfunction old_function this final __NAMESPACE__ namespace goto __DIR__

Perl has so many keywords I can't list them.

And those Fortran keywords I mentioned before:

ACCEPT ASSIGN AUTOMATIC BACKSPACE BLOCK BYTE CALL CHARACTER CLOSE COMMON COMPLEX CONTINUE DATA DECODE DIMENSION DO DOUBLE ELSE ENCODE END ENTRY EQUIVALENCE EXTERNAL FILE FORMAT FUNCTION GOTO IF IMPLICIT INCLUDE INQUIRE INTEGER INTRINSIC LOGICAL MAP NAMELIST OPEN OPTIONS PARAMETER PAUSE POINTER PRAGMA PRECISION PROGRAM REAL RECORD RETURN REWIND SAVE STATIC STOP STRUCTURE SUBROUTINE TYPE UNION VIRTUAL VOLATILE WHILE WRITE

Operators
The original Fortran operators had only a few precedences:

R: **
L: * /
L: + -
.eq. .ne. .lt. .le. .gt. .ge.
.not. .and. .or. .eqv. .neqv.
=

When choosing symbols and punctuation to use, we don't want any alphanumeric symbols, we should keep the same precedences and associativity, and, at first, we should only use those already existing in other languages.

The Algol60 operators were:

^
* %
+ -
< > <= >= = <>
== => ∨ ∧ ~
:= ; .

Note: % meant divide.

The modern operators were first used by C:

L: ++postfix –-postfix ()funcCall [] . ->
R: ++prefix --prefix +unary -unary ! ~ (castType) *deref &addr sizeof
L: *mult / %
L: +binary -binary
L: << >>
L: < <= > >=
L: == !=
L: &bitAnd
L: ^
L: |
L: &&
L: ||
R: ?:conditional
R: = += -= *= /= %= <<= >>= &= ^= |=
L: ,

Apart from alpha token sizeof, all these operators can be used in GrerlVy. We can find another use for dereferencing * and addressing &, but keep their precedence and associativity. And the comma , and -> operators will be reinstated.

C++ adds non-alpha tokens :: ::* .* ->*. The GCC version of C++ adds operators &&label <? >? .

The Java operators are:

L: [] ()methCall .
R: ++prefix ++postfix --prefix --postfix +unary -unary ~ ! (typeCast) new
L: * / %
L: +addition -binary +stringConcat
L: << >> >>>
L: < <= > >= instanceof
L: == !=
L: &bitAnd &boolAnd
L: ^bitXor ^boolXor
L: |bitOr |boolOr
L: &&
L: ||
R: ?:conditional
R: = += -= *= /= %= <<= >>= >>>= &= ^= |=

Groovy adds:

**
<=>
*. ?. .& .@ *.@
as
.. ..< in
?:elvis
=~ ==~

GrerlVy will replace as in new instanceof with symbols.

C# adds alpha operators ?? and => at the lowest precedence. JavaScript adds === and !== at the same precedence as ==. Ruby adds =~ and !~ at the same level as ===, plus ... &&= ||=. There's no further unique operators in PHP.

Perl 5 uses variable prefixes $ @ % & \. We could swipe $ to mean an escape even when we're not inside a GString. Perl 6 plans on adding many more operators: see the Perl 6 Synopsis 3. Once it's out, I'll use it as a source for even more symbols.

As well as those operator symbols from those programming languages, GrerlVy will also add:
!=~ !~ !in !instanceof

For integer division and modulo, returning two values:
/%

For defining parser combinators:
::= &&& ||| ???

I don't know what these will do yet, but might as well keep symbols balanced:
<<< <<<=

Special Variables
Groovy disallows names containing $, reserving them for its internal use. Because GrerlVy also needs internal variables, it needs to reserve some names, those containing underscore _. Some pre-existing names in Groovy/Java are coded with all capitals separated by underscores. GrerlVy will code them in camel case with a trailing underscore, e.g. SYMBOLIC_CONSTANT_FOR_IDEC will be coded as the easier-to-type symbolicConstantForIdec_, the trailing underscore showing it must be converted to the all-capitals name.

Names beginning with _ will be used as special variables, similar to those in Perl and Ruby. When choosing the meanings of special variables, we'll consider their use in regexes and Format strings, as well as those languages. Some examples from Ruby:
$! $@ $_ $. $& $~ $1 $2 $3 $= $/ $\ $0 $* $$ $?

The _ standing alone will refer to the result of the previous statement, similar to Python's interactive mode. Natural languages have pronouns, so should programming languages.

Numeric Formats
Formats of numbers must also be terse. C and C++ used the well-known syntax:

34 -72
34.1 -23.4 .77 34.1e6 78.9E-2
077 0xFF -0Xff
34U 24L 42UL

Java removes the suffix for unsigned numbers, but adds:

34.7F 22f 34.8d 77D 67.89e-2d
34 -72
Double.NEGATIVE_INFINITY Float.NaN

When supplying a string to a Float object, we can use a special syntax:

new Float('0xDEDEp-3')
new Float('0x.defP-3f')

We'll inline this into GrerlVy's syntax.

Groovy adds 34i 22G 34.8g. C# removes octal numbers, but adds decimal, e.g. 34.7m. We'll keep octal numbers, but require an o, as in 0o377.

Perl allows an underline _ for legibility, e,g. 1_234_567_890. We'll certainly enable that in GrerlVy.

Ruby mainly copies Perl, but adds ASCII codes and binary:

?a //Ascii code for 'a'
0b01101 //binary

Python has optional decimal parts after the decimal point, and imaginary numbers:

5. 5.e7
10j 11.1j .1j 1.j 7e5j

There's no further unique syntax in JavaScript and PHP.

One source of good ideas is the J language:

2b1001011   //base 2
2b12202102   //base 3, etc
3r5   // 3/5, similar to Scheme's rational numbers
3j5   // 3+5i, terser than Python's complex numbers
3p5   // 3*pi^5
3x5   // 3*e^5

GrerlVy will be about extreme tersity in programming. Grerlvy is a better choice of syntax for the Groovy/Grape AST.

Stress and Unstress in Computer Languages

2008-11-22T21:21:00.000-08:00

Computer languages could learn a few things from natural languages in their design...

Natural Language
Many natural languages, such as English, make a distinction between stressed and unstressed words. In general, nouns, verbs, and adjectives (incl adverbs ending in -ly) are stressed, while grammar words are unstressed.

For example: “I walked the spotty dog to the shop, quickly bought some bread, and returned home”. (I've bolded the syllables we stress during speech in this and following examples.)

We stress the nouns (dog, shop, bread, home), adjectives (spotty, quick), and verbs (walk, buy, return), and don't stress the grammar words (I, the, to, -ly, some, and). (Note: In Transformational Grammar, adverbs ending in -ly are considered to be a specific inflectional form of the corresponding adjectives.)

Examples of unstressed grammar words in English are conjunctions (and, or, but), conjunctive adverbs (while, because), pronouns (this, you, which), determiners (any, his), auxiliary verbs (is, may), prepositions (to, on, after), and other unclassed words (existential there, infinitive to), as well as many inflectional morphemes (-s, -'s, -ing, -ly).

Verbs are often only half-stressed instead of fully stressed, and prepositions half-stressed instead of unstressed, depending on the surrounding context, e.g. “The teacher saw the book behind the desk.” (Here, I've bold-italicized the half-stressed words.)

English has a clear distinction between grammar words and lexical words (nouns, adjectives/adverbs, and verbs) in speech.

Many languages distinguish between lexical and grammar words in their writing systems. German capitalizes the first letter of each noun. (Dutch stopped doing this in 1948, and English in the 1700's). Japanese uses Chinese characters for nouns and many adjectives, and the Japanese alphabet for grammar words and many verbs.

When using grammar words in a lexical capacity, we stress them when speaking, e.g. “I put an 'is' followed by an 'on', before the 'desk' with a 'the' before it, to make a predicate.” And when writing, we put the grammar words we're using as lexical ones inside quotes.

Using stress and unstress to separate lexical and grammar words enables English, and probably all natural languages, to be self-referential.

Computer Languages
Virtually every computer language differentiates between lexical words and grammar words.

Assembler and Cobol used indentation and leading keywords to distinguish different types of statements, and space and comma to separate items. Like many languages after them, the limited set of keywords couldn't be used for user-defined names. Fortran introduced a simple infix expression syntax for math calculations, using special symbols (+ - * etc) for the precedenced infix operators, and ( ) for bracketing. Lisp removed the indentation and keywords completely, making everything use bracketing, with space for separation, and a prefix syntax. APL removed the precedences, but introduced many more symbols for the operators. The experimentation continued until C became widespread.

C uses 3 different types of symbols for bracketing, ( ) [ ] { }. C++, Java, and C# added < > for bracketing. C uses space and , ; . for separators, and a large number of operators, organized via a complex precedence system. Java has 53 keywords; C# has 77.

The lexical words of computer languages are clear. Classes and variables are nouns. Functions and methods are verbs. Keywords beginning a statement are imperative verbs, and in some languages are indistinguishable from functions. Modifiers, interfaces, and annotations are adjectives/adverbs. The operators (+ - * / % etc) bear a similarity to prepositions, some of them (+= -= *= etc), to verbs. And I'd suggest the tokens used for bracketing and separators are clear examples of grammar words in computer languages, being similar to conjunctions and conjunctive adverbs.

In general, computer languages use some tokens (e.g. A-Z a-z 0-9 _) for naming lexical words, and others (e.g. symbols and punctuation) for grammar. Occasionally, there's exceptions, such as new and instanceof in Java. Some computer languages use other means. Perl and PHP put a @ before all lexical words, enabling all combinations of tokens to be used for names. This is similar to capitalizing all nouns in German. C# allows @ before any lexical word, but only requires it before those which double as keywords. This is similar to quoting grammar words to use them as lexical ones in English.

Newer programming languages have different ways to use Unicode tokens in names and operators. The display tokens in Unicode fall into six basic categories: letters (L), marks (M), numbers (N), symbols (S), punctuation (P), and separators (Z). Python 3.0 names can begin with any Unicode letter (L), numeric letter (in N), or the underscore (in P); subsequent tokens can also be combining marks (in M), digits (in N), and connector punctuation (in P). Scala names can begin with an upper- or lowercase Unicode letter (in L), the underscore (in P), or the dollar sign (in S); subsequent tokens can also be certain other letters (in L), numeric letters (in N), and digits (in N). Scala operators can include math and other symbols (in S). Almost all languages have the same format for numbers, beginning with a number (in N), perhaps with letters (in L) as subsequent tokens.

Perhaps the easiest way to distinguish between lexical and grammar words in GrerlVy is to use Unicode letters (L), marks (M), and numbers (N) exclusively for lexical words, and symbols (S), punctuation (P), and separators (Z) exclusively for grammar words. Of course, we still have a difficulty with the borderline case: infix operators and prefix methods, which correspond roughly to prepositions and verbs, the half-stressed words in English. I'm still thinking about that one.

Unicode for GrerlVy

2008-11-15T01:34:00.000-08:00

In looking at how to use Unicode tokens in the GrerlVy Programming Language, the first step is elimination.

Elimination
The first 65,536 Unicode tokens, from 0x0000 to 0xFFFF, are the most commonly used ones: together they're known as the Basic Multilingual Plane (BMP). Older versions of Unicode used only this plane, and the Java and C# char types, being 16 bytes, were initially designed to work with this plane only. Unicode has subsequently expanded to over 100,000 tokens, on further planes, the extra tokens being mainly ancient scripts (e.g. Egyptian hieroglyphics) and uncommon CJK characters. Although more recent versions of Java and C# also handle tokens in the newer planes, they're clunky to use. An internationalized programming language making use of the greater range of tokens that Unicode provides would find the BMP tokens to be sufficient: the tokens in the other planes would be too unfamiliar and wouldn't add much more to the boost in expressiveness, so we can assume an internationalized programming language wouldn't use them at first.

The BMP tokens are divided into different categories with one-letter codes: letters (L), marks (M), separators (Z), symbols (S), numbers (N), punctuation (P), and other (C). Within each category are various sub-categories. The 'other' category has sub-categories control (Cc), format (Cf), private use (Co), surrogate (Cs), and unassigned (Cn). The control (Cc) tokens are the familiar ASCII control characters, 0000-001F and the DEL character 007F, plus the 32 Latin-1 additions 0080-009F. These controls are used as such by most systems so we can't use them in an internationalized programming language, except for \t, \f, \r, and \n because they're already used extensively. Nor would we use the 32, mostly invisible, Format (Cf) tokens, the 6400 Private Use (Co) tokens, or the 2048 Surrogate (Cs) tokens. There are 6359 Unassigned (Cn) tokens which we wouldn't yet use.

Each Unicode token has a directionality type, which defines whether they're displayed in a leftwards or rightwards direction. About 1050 of the assigned BMP tokens are rightwards ones, mainly characters from from the Arabic, Syriac, Thaana, and Hebrew alphabets. We can make things simple in a first cut of an internationalized programming language by ignoring these tokens. These tokens are known as strongly directional. We can still use other tokens with weak and neutral directionality, their displayed direction depending on surrounding tokens. Later on, though, we would need to cater for the effect of these weakly and neutrally directional tokens in the Unicode Bidirectional Algorithm when designing the grammar of GrerlVy.

Many Unicode tokens, about 3850 of them, can be decomposed into other Unicode tokens. Some of these, about 600, are the compatibility tokens, tokens which are only in Unicode to provide easy conversion to and from other character sets. Unicode provides for normalization, where any compatibility characters in a text can easily be converted to pure Unicode forms. We should ignore these compatibility tokens.

A first cut for our programming language should also ignore tokens without a supplied font in common computer systems, such as Windows XP and Linux.

Inspection
We now need to visually inspect the valid tokens in an easy-to-grok format. We need the Unicode Properties File, and a program to munge it into a viewable form, such as this Groovy 1.6-beta-2 script (only works for Unicode tokens up to 0xFFFF):

def wtr= new OutputStreamWriter(
         new FileOutputStream(
         new File('TheOutput.txt')), 'unicode')

def maxUCode= 0xFFFF

def allChars= [:], seqReached=null, prevCode=0
new File('UnicodeData.txt').
    splitEachLine(';'){
  if(seqReached != null){
    (prevCode..Integer.parseInt(it[0],16)).each{idx->
      if(idx <= maxUCode) allChars[idx]= seqReached
    }
    seqReached= null
  }

  def ucdRec= [
    name: it[1],
    cat: it[2],   //general category (Cn)
    canon: it[3], //canonical combining class (0)
    bidi: it[4],  //bidi class (L, AL, R)
    num1: it[6],  //numeric type (None); numeric value (NaN)
    num2: it[7],  //    ditto
    num3: it[8],  //    ditto
    mirr: it[9],  //bidi mirrored (N)
    old: it[10],  //unicode 1 name
    iso: it[11],  //ISO Comment
    upp: it[12],  //Simple uppercase mapping
    low: it[13],  //Simple lowercase mapping
    tit: it[14],  //Simple titlecase mapping
    block: Character.UnicodeBlock.of(it[0] as char),
  ]

  if(it[5]){
    def a= it[5].split(' ')
    if(a[0][0] == '<'){
      b= a[0][1..-2]
      a -= a[0]
    }else b= 'unicode'
    ucdRec.decomTyp= b
    ucdRec.decomSeq= a
  }else{
    ucdRec.decomTyp= 'none'
    ucdRec.decomSeq= null
  }
  def theInt= Integer.parseInt(it[0],16)
  if(it[1] =~ /First>$/){
    seqReached= ucdRec
    prevCode= theInt
  }else if(theInt <= maxUCode)
    allChars[theInt]= ucdRec
}

def cats= [
  L:'letter', M:'mark', N:'number', S:'symbol',
  P:'punc', Z:'separator', C:'other'
]
def subcats= [
  Cc:'control', Cf:'format', Cs:'surrogate',
  Co:'privateUse', Cn:'unassigned',
  Ll:'lowercase', Lu:'uppercase', Lt:'titlecase',
  Lm:'modifier', Lo:'other',
  Mn:'nonspacing', Mc:'combiningSpacing', Me:'enclosing',
  Nd:'decimalDigit', Nl:'letter', No:'other',
  Sc:'currency', Sm:'math', Sk:'modifier', So:'other',
  Ps:'start', Pe:'end', Pi:'initialQuote', Pf:'finalQuote',
  Pc:'connector', Pd:'dash', Po:'other',
  Zl:'line', Zp:'paragraph', Zs:'space',
]
def dirs= [
  L:'leftToRight', R:'rightToLeft', AL:'rightToLeftArabic',
  AN:'arabicNumber', EN:'europeanNumber',
  ET:'europeanNumberTerminator', ES:'europeanNumberSeparator',
  CS:'commonNumberSeparator', NSM:'nonspacingMark',
  BN:'boundaryNeutral',
  B:'paragraphSeparator', S:'segmentSeparator',
  WS:'whitespace', ON:'otherNeutrals',
  LRE:'leftToRightEmbedding', RLE:'rightToLeftEmbedding',
  LRO:'leftToRightOverride', RLO:'rightToLeftOverride',
  PDF:'popDirectionalFormat',
]
def decoms= [
  'unicode', 'font', 'compat',
  'circle', 'square', 'narrow', 'wide', 'super', 'sub',
  'initial', 'medial', 'final', 'isolated',
  'fraction', 'vertical', 'small', 'noBreak',
]

(0x0000..maxUCode).
  findAll{ allChars[it] != null &&
           !(allChars[it].decomTyp in ['compat']) }.
  groupBy{ [cat:allChars[it].cat, dir:allChars[it].bidi] }.entrySet().
  findAll{ !(it.key.cat in ['Cs','Co','Cc','Cf','Cn']) &&
           !(it.key.dir in ['R','AL']) }.
  sort{ subcats.keySet().toList().indexOf(it.key.cat) }.each{ it->
    def itKeyCat= it.key.cat
    wtr<< "\r\n${cats[itKeyCat[0]]}/${subcats[it.key.cat]}, "<<
          "${dirs[it.key.dir]}: $it.value.size\r\n"
    it.value.
      groupBy{ Character.UnicodeBlock.of(it as char) }.entrySet().
      findAll{ it.key != null }.
      sort{ it.value.size }.reverse().each{
        "${it.value.collect{it as char}.join('')}\r\n"
    }
  }


wtr.close()

The output file is grouped by category, subcategory, directionality, and finally, block: easy to view.

Analysis
The first striking feature of this data is the large number of CJK and Korean characters, something I've commented on many times before. Some further odd thoughts...

The lowercase Greek alphabet (αβγδεζηθικλμνξοπρςστυφχψω) provides many tokens that could be used in a similar way to their use in mathematics, e.g. π instead of Math.PI.

Some math symbols in the MATHEMATICAL_OPERATORS block are:

∀∁∂∃∄∅∆∇
∈∉∊∋∌∍
∎∏∐∑∔∕∖∗∘∙√∛∜∝∞∟
∠∡∢∣∤∥∦∧∨∩∪∫∮∱∲∳
∴∵∶∷∸∹∺∻∼∽∾∿≀
≁≂≃≄≅≆≇≈≉≊≋≌≍≎≏
≐≑≒≓≔≕≖≗≘≙≚≛≜≝≞≟
≠≡≢≣≤≥≦≧≨≩≪≫≬
≭≮≯≰≱≲≳≴≵≶≷≸≹≺≻≼≽≾≿⊀⊁
⊂⊃⊄⊅⊆⊇⊈⊉⊊⊋⊌⊍⊎
⊏⊐⊑⊒⊓⊔
⊕⊖⊗⊘⊙⊚⊛⊜⊝⊞⊟⊠⊡
⊢⊣⊤⊥⊦⊧⊨⊩⊪⊫⊬⊭⊮⊯⊰⊱⊲⊳⊴⊵
⊶⊷⊸⊹⊺⊻⊼⊽⊾⊿⋀⋁⋂⋃⋄⋅⋆⋇
⋈⋉⋊⋋⋌⋍⋎⋏⋐⋑⋒⋓⋔⋕
⋖⋗⋘⋙⋚⋛⋜⋝⋞⋟⋠⋡
⋢⋣⋤⋥⋦⋧⋨⋩⋪⋫⋬⋭⋮⋯⋰⋱

One modern language that uses many of these mathematical operators is Sun's Fortress. Fortress also provides sequences of ASCII keys for entering these operators. The Fortress Programmers Guide leaves it vague whether the ASCII key sequences are part of an IME for the Unicode tokens, or whether the tokens are merely a convenience to make ASCII-based programs more readable.

Some arrow symbols:

←↑→↓↔↚↛↠↣↦↮⇎⇏⇒⇔
↕↖↗↘↙↜↝↞↟↡↢↤↥↧↨↩↪↫↬↭↯
↰↱↲↳↴↵↶↷↸↹↺↻↼↽↾↿⇀⇁⇂⇃
⇄⇅⇆⇇⇈⇉⇊⇋⇌⇍⇐⇑⇓⇕⇖⇗⇘⇙⇚⇛
⇜⇝⇞⇟⇠⇡⇢⇣⇤⇥⇦⇧⇨⇩⇪

Some bracketing and quoting symbols:

〈《「『【〔〖〘〚〝
〉》」』】〕〗〙〛〞〟
‘‛“‟‹«
’”›»

I'll periodically blog some ideas on how to use these and other tokens in a new Unicode-based programming language, such as GrerlVy.

The Groovy Language Family

2008-11-11T05:33:00.000-08:00

Since its birth, the Groovy Language has advanced in both function and distribution. Here are the present members of its family...

1. Groovy Classic: Birthed by James Strachan, and grown by current despots Guillaume Laforge, Jochen Theodorou, and Jeremy Rayner, the classic version of the Groovy Language is now at version 1.5.7.

2. Grails Framework: Guillaume and Graeme Rocher began work on Grails, releasing the first version, 0.1, on 30 March 2006. Grails is now the primary use case for the Groovy Language.

3. Gant, Gradle, and the modules: Russel Winder created Gant, and Hans Dockter created Gradle, just two of many modules for the Groovy Language.

4. Grape: Beginning with beta-2 of version 1.6, Grape was bundled with Groovy. Grape enables Groovy scripts to fetch jars not available locally. By the Groovy 2.0 release, Grape will even fetch the Groovy Language system itself, and so will supercede it in importance. Slowly, people will start to refer to Groovy 2.0 as Grape, and the language/framework combo as Grape 'n' Grails.

5. GroovyScript: Originally intended by Gavin Grover to be an alternative distribution of the Groovy Language should it change its name, GroovyScript is still a placeholder project. Because there might never be any distinct name-changing event for the Groovy Language, just a gradually increasing use of the new name, GroovyScript may never be born.

Of course the Groovy Language name lives on as the family grouping of all these software technologies. Who could ever forget their genesis in that historic original announcement? Viva la Groovy!

Blurring Function/Closure Boundary

2008-10-12T03:41:00.000-07:00

In a recent post to the Groovy Language mailing list, Robert Fischer suggests some ways of blurring the lines between methods and closures. I quickly posted another way, which I'll expand on here...

In functions, all external variables are hidden, and so can be shadowed:

def a='?'
def f(s){
  def a= '!', b= 'hello'
  b + ' ' + s + a
}
assert f('world') == 'hello world!'

In closures, all external variables are exposed, and can't be shadowed:

def a='?', b= 'hello'
def c= {s->
  def a2= '!' //a can't be shadowed
  b + ' ' + s + a2 //b from external context
}
assert c('world') == 'hello world!'

Hiding Some External Context
Hiding some variables, as with functions, while exposing others, as with closures, from the external static context would be groovy for Groovy 2.0. I'm forever reusing common variable names, such as a or i or x, inside closures, then needing to go back later to change them so I can get a clean compile.

Closures could enable external variables to be hidden. Assuming => as syntax, in this example, b is exposed to the closure, while a isn't, enabling a to be shadowed:

def a='?', b= 'hello'
def c= { b=> s->
  def a= '!'
  b + ' ' + s + a
}
assert c('world') == 'hello world!'

The implied syntax for a closure would then be:

def c= { * => it ->
  ...
}

meaning everything external is exposed.

Blocks could also enable external variables to be hidden:

def a='?', b= 'hello'
try{ b=>
  def a= '!'
  assert b + ' world' + a == 'hello world!'
}

The implied syntax would then be:

try{ * =>
  ...
}

meaning everything external is exposed.

Selectively Exposing External Context
Functions (and methods) could enable external variables to be selectively exposed:

def a='?', b= 'hello'
def f(s){ b=>
  def a= '!'
  b + ' ' + s + a
}
assert f('world') == 'hello world!'

The implied syntax would therefore be:

def f( ... ){ =>
  ...
}

meaning nothing external is exposed.

Classes could also enable external variables to be selectively exposed:

def a='?', b= 'hello'
class A{ b=>
  def f(s){
    def a= '!'
    b + ' ' + s + a
  }
}
assert new A().f('world') == 'hello world!'

I think such classes would need to be implemented as inner classes.

The implied syntax would be:

class A extends ... { =>
  ...
}

meaning nothing external is exposed.

Don't Repeat Yourself
Related to this syntax addition is another related one for functions/methods and classes, which eliminates repetition.

Recursively defined functions repeat the function name inside the definition:

def factorial(s){
  if(s <= 1) 1
  else s * factorial(s-1)
}
assert factorial(5) == 120

An alternative syntax not requiring function name repetition, enabling easier refactoring for those without an IDE:

def factorial(s){ self->
  if(s <= 1) 1
  else s * self(s-1)}
assert factorial(5) == 120

The implied syntax could therefore be:

def f( ... ){ self->
  ...
}

where self is the default name if no other is given.

Classes could also use this technique, with an implied syntax:

class A extends ... { this, super ->
  const(){ ... }
  ...
}

where this and super are used, as at present, if no other names are given. The const keyword, still an unused keyword in Groovy, could be used for constructors to avoid further needless repetition in Groovy.

These are just some quick ideas for future Groovy syntax.

The Groovy Grammar

2008-09-30T02:45:00.000-07:00

In my previous post, I talked about how I tried to define Groovy's Grammar declaratively using combinator parsers, and some of the problems I came across. In a perfect world, here's how I'd like to specify the grammar, rather than the present Antlr spec, so programmers can configure the syntax easily...

Lexing
We would define all the operators...

operators= [
  '(', ')', '[', ']', '{', '}',
  '?', ';', ':', ',', '.', '$', '->', '...', '?:', '=',
  '!', '||', '&&', '<=>', '==', '!=', '>=', '>', '<=', '<', 
  '++', '--', '+', '+=','-', '-=',
  '*', '*=', '/', '/=', '%', '%=', '**', '**=',
  '<<', '<<=', '>>', '>>=', '>>>', '>>>=',
  '~', '^', '^=', '|', '|=', '&', '&=',
  '*.', '?.', '.&', '@',
  '..', '..<', '=~', '==~',
]

...without telling the lexer to try '>' '>' if '>>' is rejected by the parser. The lexer/parser should be clever enough to make the lexer backtrack to other combinations if the parser rejects some syntax. (I'm using a Groovy-ish pseudocode for this blog post.)

We would also define the keywords:

keywords= [
  'class', 'enum', 'interface', 'extends', 'implements',
  'def', 'throws', 'const', 
  'boolean', 'char', 'byte', 'short', 'int', 'long',
  'float', 'double', 'void', 
  'true', 'false', 'null', 'this', 'super', 
  'new', 'as', 'in', 'instanceof', 
  'public', 'protected', 'private', 'static',
  'abstract', 'final', 'native', 'transient', 'volatile',
  'synchronized', 'strictfp', 'threadsafe', 
  'if', 'else', 'switch', 'case', 'default', 'while', 'for', 
  'break', 'continue', 'return', 'throw', 'assert', 
  'try', 'catch', 'finally', 'package', 'import', 
]

Even better is if the lexer can extract the operators and keywords automatically from the parser. We should only need to specify each lexing or parsing concept once only.

We would define whitespaces without needing to track the newlines because the parser should be able to access the source file positions of tokens:

newline= ('\r\n' | '\r' | '\n' | '\uffff')
singleLineComment= ('//' | '#!') lazy(_)* newline
multiLineComment= '/*' & lazy(_)* '*/'
space= ' ' | '\t' | '\f' | ('\' newline)
whitespaces= [singleLineComment, multiLineComment, space]

When defining GStrings, we would call the parser within the string to see where the embedded code ends, to avoid putting any parsing info within the lexer. It would also work recursively:

hexDigit= ('0'..'9'|'A'..'F'|'a'..'f')
stringEscape= '\'
  ('n' | 'r' | 't' | 'b' | 'f' | '"' | "'" | '\' | '$' |
  ('u' hexDigit.times(4)) |
  ('0'..'3' ('0'..'7').times(1,2)) |
  ('4'..'7' ('0'..'7')? )
stringChar= ! ('"' | "'" | '\' | '$' | newline)
multilineStringChar= ! ('"' | "'" | '\' | '$')

gString= '$' {data->
  Parser.parse(closureStatement, lexer, data) | Parser.parse(pathExpression, lexer, data)
  yield data
}

regexSymbol=
  ( ( ! ('*' | '/' | '$' | '\' | newline)) | ('\' '/') | ( ! ('\' newline)) ) ('*')*

lexedString=
  ("'''" lazy(multilineStringChar | stringEscape | '"' | '$' )* "'''") |
  ("'" lazy(stringChar | stringEscape | '"' | '$' )* "'") |
  ('"""' lazy(multilineStringChar | stringEscape | '"' | gString)* '"""') |
  ('"' lazy(stringChar | stringEscape | '"' | gString)* '"') |
  ('/' lazy(regexSymbol | ('\' '&') | gString)+ '/')

If a regex doesn't lex, the lexer should automatically deduce it could be operator / or /= without us telling the lexer here.

The rest of the lexing:

letter= ( 'a'..'z' | 'A'..'Z' | '_' | '\u00C0'..'\uFFFE' ) && !'\u00D7' && !'\u00F7'
digit= '0'..'9'
lexedIdentifier= (letter (letter | digit)*) | keyword
    //accept keywords rejected by parser as valid identifiers

exponent= ('e'|'E') ('+'|'-')? ('0'..'9')+
floatSuffix= 'f'|'F'|'d'|'D'
bigSuffix= 'g'|'G'

lexedNumber=
  '0' ( (('x' | 'X') & hexDigit+ ) |
      ('0'..'7')+ |
      (('0'..'9')+ ('.' ('0'..'9') | exponent | floatSuffix )) ) |
  ('1'..'9') ('0'..'9')* (
      ('l' | 'L' | 'i' | 'I') |
      bigSuffix |
      ( '.' ('0'..'9')+ exponent? (floatSuffix | bigSuffix)? ) |
      ( exponent? (floatSuffix | bigSuffix)? ) |
      floatSuffix )

lazyTokens= [keywords, stringLiteral, regexLiteral, ident, number]

lexer= { Parser.lexer(whitespaces, operators, lazyTokens) }

Parsing
The parser here is for some beta of version 1.5 of the Groovy Language, including future Groovy features currently in the parser but not yet part of the language.

The lexer returns operators, keywords, identifiers, string literals, various types of numbers (i.e. ints, longs, BigIntegers, floats, doubles, and BigDecimals), and fully-parsed GStrings.

We begin our parser definitions, firstly types (classOrInterfaceType is used now, but defined later):

builtInType=
  'boolean' | 'char' | 'byte' | 'short' | 'int' | 'long' | 'float' | 'double' | 'void'
builtInTypeSpec=   builtInType ( '[]' )*
classTypeSpec= classOrInterfaceType ( '[]' )*
typeSpec= classTypeSpec | builtInTypeSpec
typeArguments=
  '<' ( (classTypeSpec | builtInType '[]'+ ) |
        ('?' (('extends' | 'super') classOrInterfaceType)? )
      ).join(',')* '>'
classOrInterfaceType= ( lexedIdentifier typeArguments? ).join('.')

Next we define parameter declarations (annotation and expression defined later):

parameterDeclaration=
  ('def' | 'final' |annotation)* typeSpec? '...'? lexedIdentifier ('=' expression)?
parameterDeclarationList= parameterDeclaration.join(',')*

Continuing on to primary expressions (blockBody, primaryExpression, and strictContextExpression defined later):

closableBlock=
  '{' (parameterDeclarationList '->' | '->' | {return 'it' '->'}) blockBody '}'

argument=
  ((lexedIdentifier | primaryExpression) ':' | '*' ':'? )? strictContextExpression
argList= argument ((';' strictContextExpression)+ | (',' argument)* )

newExpression=
  'new'  typeArguments?  (classOrInterfaceType | builtInType)
  ( '(' argList ')' closableBlock? | ('[' expression? ']')+ )

parenthesizedExpression=
  '(' strictContextExpression.join(';')* ')' //preserves parens for AST

primaryExpression=
  'this' | 'super' | 'true' | 'false' | 'null' |
  lexedNumber | lexedString | gString |
  lexedIdentifier |
  builtInType |
  parenthesizedExpression |
  closableBlock |
  newExpression |
  '[' (argList | ':') ']'

We define a path expression (compoundStatement defined later):

pathTail=
  lexedIdentifier | lexedString | gString | parenthesizedExpression | compoundStatement
pathElement=
  ('*.' | '?.' | '.&' | '.') typeArguments? '@'? pathTail |
  '(' argList ')' |
  closableBlock |
  '[' argList ']'
pathExpression= primaryExpression pathElement*

The parser should enable us to specify simple operator precedences easily, and more complex stuff without much difficulty:

expressionPrecedences= [
  postfixExpression: { pathExpression ('++' | '--')? },
  unaryExpressionNotPlusMinus: {
    ('~' | '!') unaryExpression |
    ('(' builtInTypeSpec ')' unaryExpression |
    '(' classTypeSpec ')' unaryExpressionNotPlusMinus |
    postfixExpression
  ),
  unaryExpression: { ('++' | '--' | '-' | '+') unaryExpression | it },
  powerExpressionNotPlusMinus: { unaryExpressionNotPlusMinus ('**' it)* },
  powerExpression: { unaryExpression.join('**')+ },
  multiplicativeExpression: {
    ('++' | '--' | '+' | '-') powerExpressionNotPlusMinus
    ( ('*' | '/' | '%') it )*
  },
  additiveExpression: { it.join('+' | '-')+ },
  shiftExpression: { it.join('<<' | '>>' | '>>>' | '..' | '..<')+ },
  relationalExpression: { it
    ( (('<' | '>' | '<=' | '>=' | 'in') it) | 'instanceof' typeSpec | 'as' typeSpec )?
  },
  equalityExpression: { it.join('!=' | '==' | '<=>')+ },
  regexExpression: { it.join('=~' | '==~')+ },
  andExpression: { it.join('&')+ },
  exclusiveOrExpression: { it.join('^')+ },
  inclusiveOrExpression: { it.join('|')+ },
  logicalAndExpression: { it.join('&&')+ },
  logicalOrExpression: { it.join('||')+ },
  conditionalExpression: { it ( '?' expression ':' conditionalExpression )? },
  assignmentExpression: { it
    ( ( '=' | '+=' | '-=' | '*=' | '/=' | '%=' | '**=' |
        '<<=' | '>>=' | '>>>=' | '&=' | '^=' | '|=')
      assignmentExpression )?
  },
]

expression= Parser.buildOperatorTable(expressionPrecedences, pathExpression)

We build the toplevel elements (statements and typeDefinitions defined later). We supply a closure to implement parsing a group of statements without explicit semicolons separating them (a more readable solution than littering the parse code with 'nls'):

bareIdentifier= lexedIdentifier.join('.')*
annotationMemberValueInitializer= conditionalExpression | annotation
annotation= '@' bareIdentifier ( '('
  ( annotationMemberValueInitializer |
    (lexedIdentifier '=' annotationMemberValueInitializer).join(',')* )?
  ')' )?

modifier= 'private' | 'public' | 'protected' | 'static' | 'abstract' | 'final' |
  'native' | 'transient' | 'synchronized' | 'threadsafe' | 'strictfp' | 'volatile'
modifiers= 'def' | ('def' | modifier | annotation)+
classModifiers= (modifier | annotation)*

blockBody= (statementGroup?).join(';')*
statementGroup= {data->
  List splitData= data.split{it.lineNumber}
  List statements= []
  splitData.each{line->
    parsed= Parser.parse(statement, lexer, line)
    if( (parsed.FailedParseAtEndOfLineException && /*line is last*/)
        || parsed.FailedParseBeforeEndOfLineException ){
      /*add line to end of previous line
        and resume parsing that one from yielded location
      */
    }else if(parsed.FailedParseAtEndOfLineException){ // && line not last
      /*add to beginning of following line*/
    }else
      statements << parsed
  }
  return statements
}

compoundStatement= '{' blockBody '}'
throwsClause= 'throws' bareIdentifier.join(',')+

fieldDefinitions= (lexedIdentifier ('=' expression)? ).join(',')*
methodDefinition= (lexedIdentifier | lexedString)
  '(' parameterDeclarationList ')' throwsClause? compoundStatement?
constructorDefinition= lexedIdentifier '(' parameterDeclarationList ')'
  throwsClause? '{'
  ((typeArguments? ('this' | 'super') '(' argList ')' (';' blockBody)? ) | blockBody)
  '}'

declaration=
  ( (('def' | modifiers | typeSpec ) | (modifiers | typeSpec))
    (fieldDefinitions | methodDefinition) ) |
  (lexedIdentifier && { Character.isUpperCase(it[0]) }) fieldDefinitions
classField= typeDefinition | constructorDefinition |
  'static' compoundStatement | compoundStatement | declaration
typeParameters=
  '<' (lexedIdentifier ('extends' classOrInterfaceType.join('&')+ )? ).join(',')* '>'

classDefinition= 'class' lexedIdentifier typeParameters?
  ('extends' classOrInterfaceType)?
  ('implements' classOrInterfaceType.join(',')+ )?
  '{' (classField? ).join(';')* '}'
interfaceDefinition= 'interface' lexedIdentifier typeParameters?
  ('extends' classOrInterfaceType.join(',')+ )?
  '{' (typeDefinition | declaration ).join(';')* '}'

annotationField= typeDefinition |
  (modifiers? typeSpec)
  ( lexedIdentifier '(' ')' ('default' annotationMemberValueInitializer)? |
    fieldDefinitions | methodDefinition )
annotationDefinition= '@' 'interface' lexedIdentifier '{'
  annotationField.join(';')* '}'

enumMethodDefinition= lexedIdentifier '(' parameterDeclarationList ')'
  throwsClause? compoundStatement?
enumConstantField=
  ( typeDefinition | modifiers? typeParameters? typeSpec
    (enumMethodDefinition | methodDefinition | fieldDefinitions) ) |
  compoundStatement
enumConstant= annotation* lexedIdentifier
  ('(' argList ')')?
  ('{' enumConstantField.join(';')* '}' )?
enumDefinition= 'enum' lexedIdentifier
  ('implements' classOrInterfaceType.join(',')+ )?
  '{' (enumConstant.join(',')+ ','? )?
  (';' classField.join(';')+ )? '}'

typeDefinition= classModifiers
  (classDefinition | interfaceDefinition | enumDefinition | annotationDefinition)

Finally, the statements:

compatibleBodyStatement= compoundStatement | statement
singleDeclaration= (modifiers typeSpec? lexedIdentifier) | (typeSpec lexedIdentifier)
branchStatement= 
  'return' expression? | ('break' | 'continue') lexedIdentifier? |
  'throw' expression | 'assert' conditionalExpression ((',' | ':') expression)?
strictContextExpression=
  (singleDeclaration ('=' expression)? ) | expression | branchStatement | annotation
closureList= ( strictContextExpression? ).join(SEMI)*
forInClause=
  (singleDeclaration | lexedIdentifier) ('in' shiftExpression | ':' expression)
casesGroup= (('case' expression | 'default') ':')+ statement.join(';')+
exceptionHandler= 'catch' '(' parameterDeclaration ')' compoundStatement

statement=
  typeDefinition |
  declaration |
  expression |
  'import' 'static'? lexedIdentifier ('.' lexedIdentifier)*
      ('.' '*' | 'as' lexedIdentifier)? |
  lexedIdentifier ':' statement |
  'if' '(' conditionalExpression ')' compatibleBodyStatement
      ('else' compatibleBodyStatement)? |
  'for' '(' (closureList | forInClause) ')' compatibleBodyStatement |
  'while' '(' strictContextExpression ')' compatibleBodyStatement |
  'switch' '(' strictContextExpression ')' '{' casesGroup* '}' |
  'try' compoundStatement exceptionHandler* ('finally' compoundStatement)? |
  'synchronized' '(' strictContextExpression ')' compoundStatement |
  branchStatement

compilationUnit=
  (annotation* 'package' bareIdentifier ';')? statement.join(';')* Parser.EOF

parser= {data-> Parser.parse(compilationUnit, lexer, data) }

Summary
When the lexing and parsing is declarative, it's easy to embed closures building the AST directly in the parse, for example, in a JParsec-based Groovy lexer/parser I worked on. The more complex stuff should be handled by AST visitors afterwards.

Programmers want to define their own syntax for programming languages, just as they already configure their own settings for an IDE or OS. To do so easily, they need a declarative lexer and parser: not bottom-up parsing, but recursive descent. But the ones I tried were very slow, often bloated the generated code, didn't handle left-recursion, didn't allow the parser to backtrack to the lexer, didn't let me hook in extra parsing passes, and required me to repeat concepts. Where is that parsing technology that lets me quickly generate Groovy AST nodes using the language syntax I quickly wrote a parser for?

Grerl-Vy Ramblings

2008-09-28T08:29:00.000-07:00

Programming languages are often placed in a rough hierarchy, with system languages such as C and Java near the bottom, and others further up. Perhaps we can further subdivide the others into system-dependent "scripting" languages, and system-independent "dynamic" languages. System-dependent languages include Perl (for Unix) and Visual Basic (for Windows). Though Perl has been ported, it doesn't really work on other systems. Groovy (for the JVM) and VB.NET (for the CLR) are more recent languages, each dependent on a VM system. System-independent languages include the big three: Python, Ruby, and JavaScript. Python's been ported to different machines, and to the JVM and the CLR. JRuby has been released. They keep the same syntax and semantics irrespective of target platform. However, JavaScript has system-dependent aspects, allowing system calls to the JDK from Rhino and to the DOM from the browser-hosted versions. In a twist, the syntax for a lower-level language (Java) is used by Google's GWT to generate source for a higher-level language (JavaScript) to abstract away the browser-dependent portions.

Recently, I figured maybe Groovy could improve its fortunes by moving into the system-independent category: I looked at the feasibility of porting Groovy to .NET. I'd also received a not-too-subtle hint that with my knowledge of Groovy's internals above the AST layer, I'd be able to easily write a version that sits above the Microsoft DLR. Who was behind all that, I wonder? Anyway, I figured such a port should use the .NET libraries rather than trying to duplicate the semantics of Groovy/Java's. Similarly, a port shouldn't attempt to duplicate the language semantics, but rather aim to "groovify" C# in the same way (J)Groovy groovifies Java. So I looked at C# for the first time ever. I was surprised by how much is in there. C# has much that Groovy added to Java, such as closures (called "delegates"), categories (called "extension methods"), operator overloading, easy list and map syntax, the Elvis operator, unchecked exceptions only, a foreach statement, and properties. It has much more besides, such as continuations, indexers, and multidimensional arrays. The only significant stuff not there is parts of Groovy's dynamic MOP. I couldn't really duplicate those anyway because they are the parts most likely to change in Groovy 2.0.

The Lexer/Parser
By using the coming DLR being developed by the IronPython 2.0 team, writing such a version of Groovy for .NET would be an exercise in writing a complex parser. That brings me to the reason my attempt to write an alternative lexer/parser for the (J)Groovy AST stalled in the first place: the parsing became too complex. I hadn't wanted to use Antlr because the code isn't declarative enough. I believe the lexer/parser should be the most configurable part of a programming language, being the topmost level. Originally, I tried using the JParsec combinator parser. The code worked, though much slower than Groovy's Antlr version. I managed to use JParsec's monads to lex/parse the GStrings in a user-configurable manner, by lexing and parsing the enclosed closure while still lexing the containing GString, perhaps recursively, so all the logic for detecting the end of the enclosure is in the parser, not the lexer. However, I couldn't work out how to detect implied semi-colons. Instead of littering the parsing code with "nls" tags, I hoped to put in logic whereby we appended the line following to unsuccessful parses that reach end of line, and append to the previous line unsuccessful parses that don't reach end of line. Another issue was wanting to specify the alternatives for, say, >> (whether right-shift, or two closing angle-brackets) inside the lexer, with the parser backtracking to the lexer if need be for the alternative. The lexer/parser should be the most readable and configurable part of a programming language.

Left recursion was difficult to handle. I managed to resolve the left-recursion for block statements easily enough, but got stuck when looking at paths. The only way I could program it was to put all the choices in a big switch statement inside one combinator parser, which kind of defeats the purpose of using combinator parsers in the first place. Because I was aware of very recent research showing how to make parser combinators handle left recursion, I tried out writing my own Groovy-based parser combinators, using Ken Barclay's GParsec as a starting point, hoping to port the Haskell code from the research. However, I quickly hit speed problems, and my very early versions of the code were producing over 300 closures, at 2Kb each. I couldn't use Java for a speed increase because Java doesn't have closures, another reason I looked at .NET.

But if I ported Groovy to C#, I'd just be writing a lexer/parser for C#-style code that builds a DLR-based AST. Neither the syntax nor the libraries would be compatable with (J)Groovy's. If I called it IronGroovy, someone else could clone the code and call it Groovy#, so it would quickly splinter. Perhaps Microsoft would bundle it in its samples as an example of parsing dynamic C#-style code for the DLR. Many people think Microsoft will also have a "dynamic" keyword in a future version of C#. But most importantly, C# can be sharp, but it can never be groovy, if you get what I mean. Groovy is for the JVM, filling in the many gaps Java has failed to fill.

Refocus
So this aspect of the Grerl-Vy project has stalled, waiting for:

recursive-descent parsing technology to mature further, with recent developments being incorporated into Java-based software. If and when closures are introduced into Java 7, perhaps it'll also bundle a respectable combinator parsing library, just as Scala does.

the Groovy AST to have a well-defined specification, both its interfaces and its visitation states. Groovy could become the Java platform's answer to Microsoft's DLR. The Swing Framework was successful as a loosely-bound collection of software technology, while Grails's other leg, Groovy, is presently untangleable and under-specified.

someone to translate the names in the most common Java packages from English into simplified Chinese. Interestingly, all Chinese documentation I've seen on the web and in Chinese bookshops translate what the classes and methods do, but not the names themselves.

For now, I'm picking up where I left off 18 months ago in creating an IME to easily enable non-Chinese speakers to enter all the Unicode tokens. And because my Chinese language learning has slackened during that time, I'm refocusing on that too. When someone translates those names into Chinese and creates a programmer-configurable lexer/parser for the Groovy AST, I want to be ready for the onslaught. My holiday in NZ away from Grerl-Vy and Groovy was great in allowing me to refocus on the original vision of the Grerl-Vy project, itself a composite of two visions. The Grerl vision is to utilise all Unicode tokens in programming; the Vy vision is to stay close to the Groovy Language technology. Grerl-Vy aims to bring the full power of the Unicode Character Set to the Groovy Programming Language.

Mass-Parity-Distance Invariance

2008-09-06T22:09:00.001-07:00

During July and August, I took a break from China and Tesol and Groovy, visiting my home country, New Zealand. I hoed into a copy of Roger Penrose's The Road to Reality, and came up with an idea to explain Dark Energy...

Negative Mass
Negative mass is usually defined in such a way that Einstein's equivalence principle still holds, where gravitational mass is proportional to inertial mass. This results in some bizarre effects. But while reading Penrose's book, I got an idea on how to define negative mass so that all the positive matter and all the negative fly off in two opposite directions at the Big Bang, with the equivalence principle still holding.

The key is how we calculate the (scalar) distance with respect to some mass. For positive matter, we would continue to use the positive solution to the formula where we square root the sum of the squares of the three spatial coordinates. But we'd introduce an invariance, known as the Mass-Distance Invariance, where we'd use the negative solution to the square root for scalar distances measured with respect to negative masses.

Some consequences of this invariance are:

The same vector values for velocity and acceleration would be used for negative mass as for positive mass, but their scalar values would depend on whether positive matter was referenced, or negative matter. Negative matter would use negative speeds and, to indicate increasing speeds, negative acceleration values.

A positive-valued g-force (created by positive matter) would still mean attraction for positive matter, but repulsion for negative matter. However, a negative g-force (created by negative matter) would mean attraction for negative matter, but repulsion for positive.

When calculating the (scalar) gravitational force between two objects, the square of the distance between them would always be positive, but a positive force is attraction, and a negative force is repulsion. This means two negative masses attract, as do two positive masses, but positive and negative masses repel each other.

Such scalar values for force involving negative matter would use negative distance again when calculating energies, resulting in negative energies. Penrose mentions negative energies mess with quantum mechanical calculations, but in the real Universe, this would be OK because positive and negative energies would be partitioned off due to the gravitational effects of the Big Bang.

Therefore, when calculating scalar values in the negatively-massed side of the Universe, we'd use (1) negative distances, (2) multiplied by positive time to give negative-valued speed, (3) multiplied by positive time to give negative acceleration values to indicate increasing speeds, (4) multiplied by negative mass to give positive-valued scalar forces to indicate attraction, (5) multiplied by negative distances to give negative values for energy.

Picturing All This
When picturing such a scenario using the common "matter bends space which moves matter" 2D curved-space picture to model the 3+1D reality in general relativity, the positive matter would be on top of the sheet sinking downwards as before, but the negative matter would be under the sheet, to indicate negative distances, floating upwards, to indicate the negative mass. We can then visualize positive and negative matter each self-gravitating, but repelling each other.

Gravitons
The positive matter would act via left-handed gravitons as before, but the negative matter would act via right-handed gravitions. Penrose, in his description of Twistor Theory, says that there's a problem in the calculations getting left-handed and right-handed gravitons to interact with each other to enable graviton plane polarization, similar to what's possible with electromagnetism. But in my theory, it would be a requirement that left-handed and right-handed gravitons don't interact in any way. This enables both attractive gravity and repulsive gravity to operate at different scales in the same spacetime.

This graviton-handedness has a counterpart in neutrinos, reponsible for the vast excess of matter over antimatter in the observable Universe. So we need to follow the lead of Charge-Parity-Time (CPT) Invariance, and likewise introduce parity invariance, resulting in what I'm now calling Mass-Parity-Distance Invariance, or MPD-invariance.

Dark Energy
Observational evidence of such MPD-invariant negative matter would be an expected after-effect of the inflation of the very early Universe. The modified version of the Big Bang is that the Universe's overall zero energy fractures into equal Planck-distance-separated positive and negative amounts in the first quantum instant of the Universe, then their respective gravitational fields repelled the positive and negative away from each other, resulting in a Big Bang in two different directions along one spatial axis. The actual reason for the Big Bang can therefore be explained by quantum effects.

After the faster-than-light inflation stopped, the right-handed gravitons from the negative matter would be travelling towards the positive matter at the speed of light only, resulting in a time lag between inflation ending and the gravitational repulsion of the negative mass beginning to affect the positive mass with a renewed expansion. This is exactly what happened after about 10 billion years, what's called Dark Energy.

Negative-Frequency Electromagnetism
The photon would behave differently to the graviton. Planck's famous equation states photon energy equals Planck's constant multiplied by the frequency. Negative-energy photons would then have negative frequency, but for a photon this is not the same as changing the handedness (helicity), because photons have both electric and magnetic vectors. Both left-handed and right-handed photons have positive energy, and can polarize. Photons of negative energy/frequency, whether left-handed or right-handed, would have their electric and magnetic vectors swapped around.

Antimatter
Negative matter and antimatter are two separate concepts. Matter and antimatter created from positive energy in normal particle interactions would both have positive mass, similarly negative mass for negative energy. But a virtual particle-antiparticle pair in a vacuum would not only have an overall charge of zero, but also an overall energy of zero, one of the pair having positive energy, the other, negative. Perhaps the particle has negative mass, or perhaps the antiparticle does. This fact could provide a solution to the "hierarchy problem", there no longer being any need for supersymmetric particles to adjust quantum energy values.

The first quantum event of the Big Bang would determine how much energy, positive or negative, is in each side of the Universe. The left-handed gravitons and left-handed neutrinos go one way, their right-handed counterparts, the other. So one half of the Universe is matter with positive mass, the other half, antimatter with negative mass. One spatial dimension of the Universe is thus different to the other two, with homogeneity and isotropy being more local effects.

An alternative shape of the Universe is a four-partitioned one, where positive matter, positive antimatter, negative matter, and negative antimatter fly off in 4 different directions on a plane. This can be visualized with the 2-D saddle-shape for a hyperbolic Universe, with positive matter on top of the sheet, its matter going one way and its antimatter the other, both down the saddle on each side, and negative matter underneath the sheet, its matter and antimatter each flying off up the saddle, at ninety-degree angles to the positive matter and antimatter.

It's been two decades since I finished my undergrad maths degree, and I haven't used it since, so I'm rusty. And although I basically followed the maths in Penrose's book, I didn't get all the intricacies of manifold calculus and bundles and Langrangians. If anyone out there fills my wordy explanation of MPD-Invariance with numbers, let me know if it works or if it's rubbish. But there's more follow-on ideas I've had...

The Universe as Two Complex Planes
There's an eiry similarity between the well-known Charge-Parity-Time (CPT) Invariance and my proposed Mass-Parity-Distance (MPD) Invariance. I think it suggests a certain structure to the Universe alluded to by Penrose in his Twistor Theory. He suggests the Universe can be modelled as three complex planes (i.e. 6 real dimensions), the "imaginary" dimension being as physically real as a "real" one. But elsewhere Penrose says if there are only 4 observational dimensions of spacetime, we shouldn't try to model them with 11 or 26 dimensions. I'd suggest the Universe can be modelled as only 2 complex planes to match the 4 observational dimensions of spacetime. The extra 2 dimensions required by Penrose's model could come from the fractal dimensions created by those 2 complex planes.

A curve on a complex plane usually has a (Hausdorff) dimension of 1, but fractal curves have a dimension higher than 1, but less than or equal to 2. Only very special fractals, such as the Mandelbrot set and Julia sets, have Hausdorff dimension of 2. If there exist on any complex plane an (aleph-zero-)infinite number of concentric Hausdorff-dimension-2 sets, then I suspect the plane itself would have Hausdorff dimension 3. The union into a manifold of two such complex planes would have Hausdorff dimension 6, while only having topological dimension 4, thus satisfying Penrose's minimal number of dimensions to model our Universe.

We can create such an arrangement on both our complex planes by relating them together using an uncertainty relation. Because the Mandelbrot and Julia sets are the only sets I know of with Hausdorff dimension high enough to be valid in this model, I'll use the Mandelbrot set as an example. The basic set is only one connected curve on the complex plane, but when a computer calculates it, many circles of various colors are usually displayed to reflect different accuracies of calculation. These circles are concentric. Although only the infinitely accurate Mandelbrot set normally has any mathematical significance, when relating two complex planes together in an uncertainty relationship, the curve generated from each accuracy level takes on significance.

Relationship Between CPT and MPD Invariances
Mass would be modelled as one of the fractal dimensions, while charge modelled as the other. The two invariances, CPT and MPD, both of them having parity (i.e. space reflection) included, bear a vague resemblance to the requirements for 2n-D real manifolds to be treated as n-D complex manifolds under the Newlander-Niremberg theorem, in this case 4 real dimensions as 2 complex planes. One plane, required to be CPT-invariant, would have time as one dimension, say, the real. The imaginary dimension would be a dimension of space, and the fractal dimension, charge. The other complex plane, required to be MPD-invariant, would have the other two dimensions of space for its real and imaginary dimensions, and mass for its fractal dimension.

Planck's constant defines the uncertainty relationship between time (i.e. the real dimension of one complex plane) and energy (i.e. a proxy for mass, the fractal dimension of the other complex plane). This would be the uncertainty relationship that makes the complex planes have (Hausdorff-)dimension-3.

The other dimensionless constants of nature could be interpreted as observational coordinate mappings between dimensions on these two complex planes. The speed of light is a mapping between the time and space dimensions on the same plane. Newton's gravitational constant is a mapping between the (fractal) mass dimension and a space dimension. Coulomb's constant is a mapping between the (fractal) charge dimension and space dimension. The three space dimensions wouldn't need mapping between one another, as their differences from one another are only apparent in the helicity of the graviton and neutrino. So the four dimensionless constants would be sufficient mappings for the two planes.

Everyday Observation
I've ignored the forces without an infinite range (the strong and weak forces) in this model. The basic difference between MPD-invariant gravity and CPT-invariant electromagnetism is that in gravity, like masses attract while unlike ones repel, whereas in electromagnetism, like charges repel while unlike ones attract. The logical effect of this (ignoring finite-range forces) is that gravity's masses are real numbers, while charges are polar.

So we have two complex planes, each with three dimensions, i.e. real-imaginary-fractal. The first has Time-Distance-Charge, the second, Distance-Distance-Mass. Perhaps, in our own everyday observation of these planes, the charge, having polar (i.e. 0 or +1 or -1) values only, doesn't require its total dimensional freedom to operate, and only needs a Planck-distance portion of the (fractal) charge and (imaginary) distance dimensions. So the second complex plane "takes" the excess distance dimension from the first plane to create 3D-space, and the mass, having aggregative values, also "takes" the excess fractal freedom of the charge. So we end up with 1D-time, 3D-space, polar-charge, and aggregative-mass.

If I had the time, I'd be looking at the maths for relating two complex planes together, each with (Hausdorff-)2D fractal curves, using an uncertainty relationship, trying to derive relativity axioms and asymmetric time and such stuff. But I've now got other demands on my time, hence this blog entry. If my description rings a bell with anyone, let me know how it goes.

Word Classes in English and Groovy

2008-06-20T23:38:00.000-07:00

When I was in primary school, I learnt that English had 8 parts of speech: nouns, verbs, adjectives, adverbs, pronouns, conjunctions, prepositions, and articles. Nowadays linguists call them word classes. Since working in Tesol, I've learnt that words in English are better classified as falling somewhere along a continuum, with conjunctions, the most grammatical words, at one end, and proper nouns, the most lexical, at the other.

We'll take a quick look at these word classes in English grammar, then look at the similar concept in the Groovy Language. (Note: The English grammar is very simple, and based on what I remember from personal reading, not academic study, so I don't guarantee total correctness).

Word Classes in English

Conjunctions
The most grammatical words in English are and, or, and not, the same operators as in propositional logic. and and or can be used to join together most words anywhere further along the continuum. Most obvious are the lexical words, e.g:
  the book, the pen, and the pad (nouns)
  black and blue (adjectives)
  slowly and carefully (adverbs)
  to stop, listen, and sing (verbs)
  to put up or shut up (phrasal verbs)

Also, multiword lexical forms, such as phrases and clauses, can be similarly joined:
  the house, dark blue and three storeys high, ... (adjectival phrases)
  The batter hit the ball and the fielder caught it. (clauses)

But more grammatical words at the same position on the continuum can be joined:
  your performance is over and above expectations (prepositions)
  I could and should go (auxiliary verbs)
  They were and are studying (different type of auxiliary verbs)
  this and that (pronouns)

Incidentally, the continuum can have more than one type of multiword form at the same position, such as adverbials and prepositional phrases:
  They walked, very silently and with great care, ...

and and or are 2 of only 7 conjunctions in English, memorized by the acronym FANBOYS: for, and, nor, but, or, yet, and so. But and and or are more grammatical than the other five conjunctions, and can be used to join the others together, e.g:
  It was difficult, yet and so I tried.

The propositional logic operators are the most grammatical words in English.

Proforms
Next along the continuum are proforms, words that take the place of other more lexical words. In English, the most common type of proform is the pronoun, e.g. he, she, this, which also has determiner form, e.g. his, hers. For example:
  The dog chased the cat, but lost it. (pronoun: it)
  The dog escaped from the goat, but lost its collar. (determiner: its)

Other word classes and multiword forms have proforms. For example, pro-verb do/did:
  I enjoyed the film, and so did the ushers.
Gap for pro-verb:
  We found the south exit, and the other team, the north exit.
Pro-adjective such:
  We experienced a humid day, and also such a night.
Pro-adverb thus:
  Swiftly the Italians played; thus also did the Brazilians.
Proform for multiword adverbial so:
  The programmers finished totally on time; so did the testers.

Particles
Next are a large number of miscellaneous words between grammatical and lexical, which some call particles. Examples are interjections, articles (a/an/the), phrasal verb particles, conjunctive adverbs, sentence connectors, verb auxiliaries, not, only, infinitive's to, etc.

English, and I guess every natural language, is really a mess, and the particles are a way of categorizing the messy stuff.

Prepositions and Verbs
The first lexical word class along the continuum is the prepositions. In Hallidayan Functional Grammar, they're considered to be reduced verbs. Some examples: under, over, through, in. There are also multiword prepositional groups, e.g: up to, out of, with respect to, in lieu of.

Further along the continuum are the verbs, e.g. listen, write, walk. Verbs can be multiword, such as phrasal verbs, e.g. put up, shut up, prepositional phrasal verbs, e.g. get on with, put up with, and verb groups, e.g: will be speaking, has walked, to have gotten on with.

Adjectives and Nouns
Next along the continuum are adjectives, e.g. black, blacker, blackest. In Chomskian Transformational Grammar, adverbs ending in -ly are considered to be the same as adjectives, only modified at the surface level, e.g. slowly, slower, slowest.

Adjectives/adverbs can be multiword, e.g:
  The building is three storeys high. (adjectival phrase)
  That cat walks incredibly slowly. (adverb word group)

Next are common nouns, both count nouns, e.g. pen, pens, and mass nouns, e.g. coffee, hope. Nouns can be built into noun phrases, e.g. the long dark blue pen.

Just as verbs and prepositions are related, so are nouns and adjectives. Abstract ideas often only differ grammatically, e.g:
  Jack is very hopeful.
  Jack has much hope.
  Jack has many hopes.

At the lexical end of the grammar-lexis continuum are proper nouns. These can be phrases we construct from other words, e.g. the Speaker's Tavern, foreign words, e.g. pyjamas, fooyung, or even invented words, e.g. Kodak, Pepsi.

The largest word class in English are the nouns, then the adjectives, then verbs. When new words enter English, they're usually nouns. Some will become adjectives and maybe verbs, but very few ever move further along the continuum towards the grammar end. Although English has many Norman words from 800 or 900 years ago, very few are prepositions, and all the other more grammatical words came from Anglo-Saxon.

Perhaps all natural languages have a word class continuum with prepositional logic words at one end, and definable nouns at the other.

Word Classes in Groovy
Groovy uses both symbols and alphanumeric keywords for grammar, both lexed and parsed grammar. Groovy builds on Java, and hence C++ and C, for its tokens.

Bracketing and Separators
Perhaps the most grammatical along the continuum are the various bracketing symbols. Some have different tokens for opening and closing, e.g:
  /* */ ( ) [ ] { } < >
while others use the same token for both, e.g:
  """ ''' " ' /
There's no corresponding word class in English because English uses prosody (tone, stress, pause, etc) rather than words for the bracketing function.

Next along Groovy's continuum could be separators, e.g:
  , ; : ->
We can use , and ; for lists of elements, similar to and and or in English.

Groovy has a very limited repertoire of pronouns, only this and it.

Verbs and Prepositions
Perhaps operators are like English prepositions, e.g:
  == != > >= < <= <=>
  .. ..< ?: ? : . .@ ?. *. .& ++ -- + - * / % **
  & | ^ ! ~ << >> >>> && || =~ ==~
while some operators are almost like verbs, e.g:
  = += -= *= /= %= **= &= |= ^= <<= >>= >>>=

Some operators are represented by keywords in Groovy, viz. prepositions, an adjective, and a multiword noun-preposition, i.e:
  in as new instanceof

Verbs in indicative form are used in definitions, e.g:
throws extends implements
... .*

The most common verb form is the imperative, e.g:
  switch, do, try, catch, assert, return, throw
  break, continue, import, def, goto
though sometimes English adverbs are used as commands in Groovy, e.g:
  if, else, while, for, finally
Also used for this are nouns, e.g:
  case, default
and symbols, e.g:
  \ // #! $

Nouns and Adjectives
Groovy uses English adjectives for adjectival functions in Groovy, e.g:
  public, protected, private, abstract, final, static
  transient, volatile, strictfp, synchronized, native, const

Groovy has many built-in Groovy common nouns, e.g:
  class, interface, enum, package
  super, true, false, null
  25.49f, \u004F, 0x7E, 123e7

Some of them can also be used like adjectives, e.g:
  boolean, char, byte, short, int, long, float, double, void
are nouns (types) that can precede other nouns (variables), like Toy in A Toy Story.

We can define our own Groovy proper nouns using letters, digits, underscore, and dollar sign, e.g:
  MY_NAME, closure$17, αβγδε

Using @, we can also define our own Groovy adjectives.

Because Groovy is syntactically derived from Java, and hence from C++ and C, it, like English, is a little messy in its choice of tokens.

Notice also the different emphasis of word classes between English and Groovy, e.g:

Groovy uses tokens for bracketing while English uses non-token cues
English uses far more proforms than Groovy, which forces us to use temporary variables a lot
English uses Huffman coding by shortening common words like prepositions, while Groovy retains instanceof and implements

Conclusion: The Unicode Future
Unicode divides its tokens into different categories: letters (L), marks (M), separators (Z), symbols (S), numbers (N), punctuation (P), and other (C). Within each are various sub-categories. I'm looking at how best to use all Unicode characters (not just CJK ones) when extending a Java-like language such as Groovy with more tokens. The more tokens a language has, the terser it can be written while retaining clarity. Unicode's now a standard, so perhaps programmers will be more motivated to learn them than when APL was released. And modern IME's enable all tokens to be entered easily, for example, my ideas for the Latin-1 characters. Such a Unicode-based grammar must be backwards-compatible, Huffman coded, and easy to enter in the keyboard.

The Future of Programming: Chinese Characters

2008-06-13T04:17:00.000-07:00

Last year, I wrote some wordy blog entries on this subject, see Programming in Unicode, part 1 and part 2. Here's a brief rehash of them...

Deep down in their hearts, programmers want to read and write terse code. Hence, along came Perl, Python, Ruby, and for the JVM, Groovy. Lisp macros are enjoying a renaissance. But programmers still want code to be clear, so many reject regexes and the J language as being so terse they're unreadable. All these languages rely on ASCII. The tersity of these languages comes from maximizing the use of grammar, the different ways tokens can be combined. The same 100 or so tokens are used.

Many people can type those 100 tokens faster than they can write them, but can write thousands more they can't type. If there were more tokens available for programming, we could make our code terser by using a greater vocabulary. The greater the range of tokens a language has, the terser its code can be written. APL tried it many years ago but didn't become popular, perhaps because programmers didn't want to learn the unfamiliar tokens and how to enter them. But Unicode has since arrived and is used almost everywhere, including on Java, Windows, and Linux, so programmers already know some of it.

With over 100,000 tokens, Unicode consists of alphabetic letters, CJK (unified Chinese, Japanese, and Korean) characters, digits, symbols, punctuation, and other stuff. Many programming languages already allow all Unicode characters in string contents and comments, which programmers in non-Latin-alphabet countries (e.g. Greece, Russia, China, Japan) often use. Very few programming languages allow Unicode symbols and punctuation in the code: perhaps language designers don't want to allow anything resembling C++ operator overloading.

But many programming languages do allow Unicode alphabetic letters and CJK characters in names. Because there already exists agreed meanings for combinations of these, derived from their respective natural languages, programmers can increase tersity while keeping readability. However, this facility isn't used very much, maybe because the keywords and names in supplied libraries (such as class and method names in java.lang) are only available in English.

I suspect programmers from places not using the Latin alphabet would use their own alphabets in user-defined names in a certain programming language if it was fully internationalized, i.e., if they could:

configure the natural language for the pre-supplied keywords and names (e.g. in java.lang)
easily refactor their code between natural languages (ChinesePython doesn't do this)
use a mixture of English and their own language in code so they could take up new names incrementally

Most programming languages enable internationalized software and webpages, but the languages themselves and their libraries are not internationalized. Although now rare, most programming languages will one day be internationalized, following in the trend of the software they're used to write. The only question is how long this will take.

However, I suspect most natural languages wouldn't actually be used with internationalized programming as there's no real reason to. Programmers in non-English countries can read English and use programming libraries easily, especially with IDE auto-completors. Writing foreigner-readable programs in English will be more important.

To become popular in a programming language, a natural language must:

have many tokens available, enabling much terser code, while retaining clarity. East Asian ideographic languages qualify here: in fact, 80% of Unicode tokens are CJK or Korean-only characters.
be readable at the normal coding font. Japanese kanji and complex Chinese characters (used in Hong Kong, Taiwan, and Chinatowns) don't qualify here, leaving only Korean and simplified Chinese (used in Mainland China).
be easily entered via the keyboard. An IME (input method editor) allows Chinese characters to be entered easily, either as sounds or shapes. The IME for programming could be merged with an IDE auto-completor for even easier input.

And to be the most popular natural language used in programming, it must:

enable more tokens to be added, using only present possible components and their arrangements. Chinese characters are composed of over 500 different components (many still unused), in many possible arrangements, while Korean has only 24 components in only one possible arrangement.
be used by a large demographic and economic base. Mainland China has over 1.3 billion people and is consistently one of the fastest growing economies in the world.

About a year ago, I posted a comment on Daniel Sun's blog on how to write a Groovy program in Chinese. (The implementation is proof-of-concept only; a scalable one would be different.) The English version is:

content.tokenize().groupBy{ it }.
  collect{ ['key':it.key, 'value':it.value.size()] }.
  findAll{ it.value > 1 }.sort{ it.value }.reverse().
  each{ println "${it.key.padLeft( 12 )} : $it.value" }

The Chinese version reduces by over half the size (Chinese font required):

物.割().组{它}.集{ ['钥':它.钥, '价':它.价.夵()] }.
  都{它.价>1}.分{它.价}.向().每{打"${它.钥.左(12)}: $它.价"}

I believe this reduction is just the beginning of the tersity that using all Chinese characters in programming will bring. The syntax of present-day programming languages is designed to accommodate their ASCII vocabulary. With a Unicode vocabulary, the language grammar could be designed differently to make use of the greater vocabulary of tokens. As one example of many: if all modifiers are each represented by a single Chinese character, for 'public class' we could just write '公类' without a space between (just like in Chinese writing), instead of '公类', making it terser.

A terse programming language and a tersely-written natural language used together means greater semantic density, more meaning in each screenful or pageful, hence it’s easier to see and understand what's happening in the program. Dynamic language advocates claim this benefit for dynamic programming over static programming: the benefit is enhanced for Chinese characters over the Latin alphabet.

If only 3000 of the simplest-written 70,000 CJK characters in Unicode are used, there are millions of unique two-Chinese-character words. Imagine the reduction in code sizes if the Chinese uniquely map them to every name (packages, classes, methods, fields, etc) in the entire Java class libraries. Just as Perl, Python, and Ruby are used because of the tersity of their grammar, so also Chinese programming will eventually become popular because of the tersity of its vocabulary.

Furthermore, in an internationalized programming language, not only could Chinese programmers mix Chinese characters with the Latin alphabet in their code, but so could Western programmers. Hackers want to write terse code, and will experiment with new languages and tools at home if they can't in their day jobs. They'll begin learning and typing Chinese characters if it reduces clutter on the screen, there's generally available Chinese translations of the names, they can enter the characters easily, and start using them incrementally. By incrementally I mean only as fast as they can learn the new vocabulary, so that some names are in one language and some in another. This is much easier if the two natural languages use different alphabets, as do English with Chinese. A good IDE plugin could transform the names in a program between two such natural languages easily enough.

Non-Chinese programmers won't have to learn Chinese speaking, listening, grammar, or writing. They can just learn to read characters and type them, at their own pace. Typing Chinese is quite different to writing it, requiring recognizing eligible characters in a popup menu. They can learn the sound of a character without the syllabic tone, or instead just learn the shape.

Having begun using simplified Chinese characters in programs, programmers will naturally progress to all the left-to-right characters in the Unicode basic multilingual plane. They'll develop libraries of shorthands, typing π instead of Math.PI. There’s a deep urge within hackers to write programs with mathlike tersity, to marvel at the power portrayed by a few lines of code. Software developers all over the world could be typing in Chinese within decades.

Chinese character data file available...
Recently, I analyzed the most common 20,934 Chinese characters in Unicode (the 20,923 characters in the Unicode CJK common ideograph block, plus the 12 unique characters from the CJK compatibility block), aiming to design an input method easy for foreigners to enter CJK characters.

For each character, I've recorded one or two constituent components, and a decomposition type. Only pictorial configurations are used, not semantic ones, because the decompositions are intended for foreigners when they first start to learn CJK characters, before they're familiar with meanings of characters. Where characters have typeface differences I've used the one in the Unicode spec reference listing. When there's more than one possible configuration, I've selected one based on how I think a fellow foreigner will analyse the character. I've created a few thousand characters to cater for decomposition components not themselves among my collected characters. (Although many are in the CJK extension A and B blocks, I kept those out of scope.) To represent these extra characters in the data, sometimes I've used a multi-character sequence, sometimes a user-defined glyph.

The data file is CSV-format, with 4 fields:

the character
first component
either second component, or -
type of decomposition

Here's a zip of that data file and truetype font file if anyone's interested.

Base-100 Arithmetic

2008-06-04T17:35:00.000-07:00

(reposted)

In The Number Sense, Stanislas Dehaene says that in Cantonese and Mandarin, the sounds for the numbers are much shorter than in Western languages, and so native speakers of those Chinese languages can speak numbers quicker. He argues that this enables them to do mental math quicker than speakers of Western languages. In many parts of Asia including China, learning mental math is considered very important for children.

Dehaene also writes elsewhere in his book that many people who can do fast mental math not only practise the many calculation shortcuts, but also often memorize the products of 2-digit numbers. I've wondered if people memorizing such products would be better off to use a base-100 instead of base-10 system, that is, to create a hundred digits and map them to the numbers from 0 to 99. After some initial memorization, it would be easy to convert back and forth between them. Even better is if Chinese sounds were used for the base-100 digits, taking advantage of the short sounds. The Chinese group digits into groups of four, unlike English-speakers' groups of three, making Chinese numbering even more suitable.

The first ten digits already exist: 0零, 1一, 2二, 3三, 4四, 5五, 6六, 7七, 8八, and 9九. There's already characters for some of the other 2-digit numbers: 10十, 20廿, 30卅, and 40卌. Perhaps also 木 for 80 (from Chinese riddles) and 半 (meaning ½) for 50. Maybe in some cases these characters for multiples of ten could be used as radicals in associated numbers, for example, digits related in some certain way to 80 could be represented by characters with the 木 radical (eg, 相枩來枳林柬朿朾朽朳朲朰東杰, etc). There's many more existing sequences that could be used in some way, like the 10 stems (甲乙丙丁戊己庚辛壬癸), the 12 branches (子丑寅卯辰巳午未申酉戌亥), or the Yi Ching characters. What is most important, though, is that the sound of each digit from 0 to 99 be different. Because there's about 400 different sounds in Mandarin Chinese, that would be possible.

The easy part for those learning such base-100 arithmetic would be memorizing every mapping between a 2-digit base-10 number and the matching base-100 digit. Children could learn that before they're 3 years old. To do any effective mental math, they would need to memorize many sums and products of pairs of base-100 digits, far more difficult. If they memorized sums by putting the higher number first, and products by putting the lower first, they wouldn't need to remember whether a sequence of four base-100 digits was a sum or product, they would only memorize the sequence itself. If the two numbers were the same, it would be the product. This gives 5050 different ways two base-100 digits can be multiplied together and 4950 ways they can be added: 10,000 combinations in total.

Many of those 10,000, though, could be worked out using shortcuts based on patterns. For example, to multiply two numbers, such as 93 x 98, by using the complement (on 100) of each number, 7 and 2, we can calculate the complement of their sum, 91, followed by their product, 14, giving the final result 9114. This particular example is really only useful in base-10 for numbers quite close to 100, but in base-100, it can be used for all numbers over 50. At the cost of memorizing 50 pairs of complements (1+99, 2+98, etc), we can reduce the 10,000 combinations down by 1275, to 8725.

There's many other shortcuts that could be utilized to reduce that number down considerably further. I suspect those shortcuts would be based on the common divisors of 100, i.e. 2, 4, 5, 10, 20, 25, and 50. For example, when adding 25 + 22, in my mind I calculate it as 25 + (25 – 3) = (2 * 25) – 3.

Of the four-character sequences that would need to be memorized, if many of them bore some pictorial or phonetic resemblance to the thousands of four-character proverbs (成语) that Chinese children already learn by rote, they'd find it much easier to memorize them. In Chinese proverbs, only the content words are recited, not the grammar words, so English proverbs in the Chinese style would be "Stitch time, save nine", "Stone roll, no moss", "Bird hand, two bush", etc. This is what would make it far easier for native Chinese speakers to do base-100 mental math than Westerners learning such arithmetic.

Here's an example of this technique, but using an English proverb instead, with associations 13=bird, 19=hand, 2=two, and 47=bush. To multiply 13 x 19, there's no shortcut, so we'd recite the associated sounds, with the lower number first for multiplication, i.e., 13 x 19 = “bird hand”. We'd automatically finish it in our heads, i.e., “two bush” = 0247. Viola!

I don't know of any existing base-100 arithmetic in China, having never seen any websites or books on the subject. What such base-100 arithmetic needs is for a native Chinese speaker with a background in computing and linguistics to design and run the intensive computations necessary to assign the best possible mapping between 2-digit numbers and base-100 digits, so the memorizations will be easiest for native-speaking Chinese children. It would be a time-consuming input-intensive programming task with a deliverable of only 90 ordered Chinese characters. An example of the future of computing, perhaps?

Ejoty in Groovy

2008-06-03T22:28:00.000-07:00

Ejoty is a word invented by magician Stewart James to describe the mental skill of easily remembering the numeric value of each letter of the English alphabet (i.e. A=1, B=2, ..., Z=26) to enable quick mental calculation of the value of words (e.g. WORD = W + O + R + D = 23 + 15 + 18 + 4 = 60). The letters in ejoty refer to the ordered multiples of 5 in the alphabet, i.e. E=5, J=10, O=15, T=20, Y=25, which, if we memorize those, will enable us to easily calculate the values of most other letters by using only 1 or 2 offsets.

A way musical and aural learners could use to learn the values of letters is to remember the value of the first letter in each foot of the popular children's song "abcd efg, hijk lmnop, qrs tuv, wx yz", i.e. "1 5, 8 12, 17 20, 23 25".

If we can instantly know the value of each letter, we can more easily practise adding a sequence of numbers in our heads whenever we see words written down somewhere. For example, signs we see when riding public transport:
SYDNEY = 19 + 25 + 4 + 14 + 5 + 25 = 92
FLINDERS = 6 + 12 + 9 + 14 + 4 + 5 + 18 + 19 = 87

Using Groovy
To more quickly ejotize words, we can learn the sums of common letter sequences off by heart. To find the most common ones, we can write a Groovy program...

After extracting the English word list english.3 within this zip file, we can run this script:


def gramVal(gr){
  def tot= 0
  gr.each{ tot += (it as int) - 96 }
  tot
}

def grams= [:]
new File("english.3").eachLine{word->
  word -= "'"
  for(int i in 2..word.size())
    if(word.size() >= i)
      for(int j in 0..word.size() - i){
        def gram= word[j..j+i-1].toLowerCase()
        if( grams[gram] != null ) grams[gram]++
        else grams[gram]= 1
      }
}

grams.entrySet().findAll{it.value > 200}
     .sort{it.value}.reverse().each{
  def gm= gramVal(it.key)
  println "$it.key ($gm): $it.value"
}

Only the sequences of letters occuring more than 200 times in that word list will be displayed by that version of the program. The first 20 lines output are:


an (15): 2634
er (23): 2606
in (23): 2080
ar (19): 1977
on (29): 1780
te (25): 1750
ra (19): 1732
en (19): 1625
al (13): 1570
ro (33): 1498
ri (27): 1485
is (28): 1472
la (13): 1444
or (33): 1426
le (17): 1425
at (21): 1404
ch (11): 1327
st (39): 1303
re (23): 1269
ti (29): 1253

(The reason the commonly-occuring th doesn't occur is because the program doesn't consider word frequencies in normal text.)

Ejoty In Reverse
Perhaps we want to easily convert numbers to letters. We could learn the letters for the numbers up to 26 easily enough, but what if we want to convert higher numbers to something. What about to groups of letters, where their sum is the number? We'd need to generate some common possibilities. This Groovy code uses the grams list we generated in the previous code sample to generate the 5 most common sequences for numbers up to 100:


def grGrams= grams.groupBy{gramVal(it.key)}
grGrams.entrySet().findAll{it.key <= 100}
       .sort{it.key}.each{
  print "$it.key (${it.value.inject(0){flo,itt-> flo+itt.value}}): "
  def set= it.value.entrySet().sort{it.value}.reverse()
  def setSz= 5
  if(set.size() >= setSz) set= set[0..setSz-1]
  println set.collect{"$it.key($it.value)" }.join(', ')
}

Here's a segment of output showing the most common letter sequences for numbers greater than 26:


27 (10243): ri(1485), lo(973), ol(955), sh(587), ve(442)
28 (11695): is(1472), th(855), si(785), mo(709), om(691)
29 (11231): on(1780), ti(1253), it(919), no(768), ell(251)
30 (7766): oo(370), ing(362), ati(245), iu(234), sk(207)
31 (7516): op(660), po(534), rm(330), vi(294), her(217)
32 (8183): sm(361), rn(276), tic(255), eri(249), mar(188)
33 (11647): ro(1498), or(1426), ul(550), hy(403), ns(355)
34 (9931): nt(950), os(793), um(504), so(424), pr(304)
35 (9482): to(1077), ot(634), un(541), ant(321), sp(282)
36 (8867): ou(788), rr(288), pt(183), ato(165), min(165)
37 (8499): rs(330), ly(267), ov(250), yl(220), pu(149)
38 (10054): tr(740), ss(458), rt(414), ion(261), qu(256)
39 (11026): st(1303), ur(745), ent(320), ru(304), per(244)
40 (9339): us(1130), su(353), tt(308), ast(225), sta(211)
41 (9141): ut(364), tu(282), ism(252), rin(191), yp(178)
42 (8756): ers(200), mon(199), olo(170), res(144), ori(132)
43 (9499): ter(534), ry(378), tin(177), nti(151), ium(138)
44 (8233): ste(219), ys(184), est(168), tio(157), sy(107)
45 (7260): ty(229), ver(182), yt(128), tte(107), aceou(104)
46 (7459): ris(147), rom(135), mor(108), orm(105), los(77)
47 (7762): sis(206), tri(180), ron(156), rit(118), ssi(107)
48 (8292): ist(236), sti(162), uri(116), tis(115), eter(114)
49 (7790): ton(202), rop(143), ont(122), pro(120), phy(120)
50 (6711): oto(117), graph(83), oun(59), tric(57), low(54)

There's plenty of choices there.

The code uses the groupBy GDK function, and the output gives a visual representation of applying it to some data.

A Groovy-like Syntax for Scheme

2008-05-19T01:18:00.000-07:00

In the last few years, I started 3 unfinished projects...

(1) Analyze the 20,000 most common CJK tokens in Unicode, aiming to make them easier for Westerners to use in programs. I've realized they're far more complex than I first thought. The data's entered and the scripts written, but I haven't finished it because there'd be too many special cases and loose ends, just like the existing IME's.

(2) Document the Groovy Language so non-Java programmers can learn it easily. I've realized the Java libraries are quite complex, and a dynamic language on top of it, while easy for existing Java programmers, may be more difficult to learn than Python or Ruby for non-Java programmers.

(3) Write an alternative lexer/parser for the Groovy AST. First, I used JParsec, then my own combinator parsing library. I've grasped the power of closures quite well, but this programming task showed me how powerful syntactic macros would be. I've since been learning Scheme with the aim of seeing how a language for the Groovy AST could support such macros.

Although Scheme is powerful, the parentheses make it quite hard to read. Reading Groovy is easy for Java programmers, but learning to read Scheme is easy for no-one, though given enough practice I'm sure it would become easy. So I'm experimenting with giving Scheme a Groovy-like syntax, called "Ghreme".

Ghreme Syntax

Some of the syntactic features we'd need...

(1) Infix notation. Instead of:
  (+ 3 4 (/ 15 3))
we'd write:
  3 + 4 + (15 / 3)

(2) Method names outside the parentheses. Instead of:
  (print "abcdefg")
we'd write:
  print("abcdefg")

(3) The first parameter of a function to be written as an object. Why not when Groovy has multimethods? Instead of:
  (seq (alt (some (match 'a')) (seq (match 'b') (maybe (match 'c'))))
    (many (match 'd')))
we'd write:
  alt(match('a').some(),
      match('b').seq(match('c').maybe()) ).
  seq(match('d').many())

(4) Simpler notation for lists and maps. Instead of:
  '("a" 2 "c")
we'd write:
  ['a', 2, 'c']

(5) Simpler notation for Lambda functions. Instead of:
  (lambda (a b) (display (+ a b)) (newline))
we'd write:
  {a,b-> display(a + b); newline() }

After creating Ghreme, I hope I'll understand syntactic macros enough to bring them to Groovy via Grerl-Vy. Assuming I'm not blinded by Eric's light and start evangelizing Ghreme to Groovy developers.

P.S. Because my blog's now about more than Groovy, I'm shifting the "Groovy" in its name to its rightful place.

Groovy Tracks

2008-04-16T08:26:00.000-07:00

Groovy 1.5.5 has just been released, which compiles code 3 to 5 times faster than the previous release in the Groovy 1.5 version track. The changes were backported from the upcoming beta-1 in the Groovy 1.6 version track. There are now two tracks of the Groovy Language.

About six months ago I got a tip-off that some Groovy developers wanted to rename the Groovy Language after version 1.5 to shake me off as they saw me as a threat to their control of the Groovy Language brand. Also, they wanted to keep the Groovy name for the 1.5 version so they could still reap the benefits of their legacy investment in the brand. In my previous post in the Tracking Groovy series, Graeme Rocher denies this in the comment he made on the morning of 1 April: was he being naïve or devious in his timing?

Groovy 1.1 RC-2 was renumbered to 1.5.0 in a preplanned email exchange just days before its final release last year. Now I see Groovy 1.6 beta-1 will be released in a few weeks, just before the annual JavaOne 2008 conference, Groovy's big marketing event. Is this again just coincidental timing?

However the developers might rename Groovy 1.6, whether they consolidate Groovy 1.6 and Grails under one Grails brand, or whether they rebrand Groovy 1.6 to something different with a G at the front, like Grape 'n' Grails, it would be a grave loss for Groovy. Groovy's increasing uptake would slow because it would take a while to propagate knowledge of the new name throughout the Java community, and heavy marketing at JavaOne wouldn't be enough. The Groovy brand would still live on anyway because many people, myself included, would embed Groovy 1.6 in some product with “Groovy” in its name, because we'd then be allowed to. However, the Groovy brand might splinter.

And I'd have to manage the Groovy Language brand myself, something I don't want to do because the Groovy Developers already do such a good job at it, far better than I could.

Syntactic Rank in English and Groovy

2008-04-09T05:07:00.000-07:00

There's many similarities between natural languages and computer languages. Let's look at one aspect of language, syntactic rank, and compare it in English and Groovy. The analysis of English is basic only, just enough so I can compare it to Groovy.

Rank in English
In English, structures larger than a sentence are different in written and spoken English: writing has paragraphs and speech has exchanges. Sentences are strung together into a cohesive sequence by pronoun linking, transition phrases, etc. Syntax kicks in below the sentence-level by defining 5 ranks: (1) sentence, (2) clause, (3) phrase, (4) word, (5) morpheme. Items at one rank are composed of items from the next rank down, with the morpheme being the lowest, atomic, rank.

A sentence can be simple, consisting of only one clause, or more complex. For example, this compound-complex sentence:

The batter hit the ball hard, but, because the wind blew it his way, the out-fielder caught it easily.

consists of 3 clauses, in a tree-like manner:

(the batter hit the ball hard)   //compound constituent
but
( because
  (the wind blew it his way)   //complex constituents
  (the out-fielder caught it easily)
)

A clause consists of phrases (sometimes called groups). For example:

Surprisingly, the batter has slammed the ball out of the pitch.

has the tree structure:

surprisingly   //adverb phrase
( the batter   //noun phrase
  ( has slammed   //verb phrase
    the ball   //noun phrase
    (out of the pitch)   //prepositional phrase
  )
)

A very common structure of clause is noun phrase followed by predicate. For example:

The beekeeper became a mountain climber.

is divided into:

the beekeeper //noun phrase at the head, called a subject
became a mountain climber
//predicate, which can be further broken down

A phrase consists of words. For example, this noun phrase:

A big bright red truck

has structure:

a
big
(bright red) //not a word, but an example of rank-shifting
truck

The (bright red) isn't a word, but another phrase used as a word. This is called rank-shifting. Sometimes, a rank can be shifted more than one place. For example:

the pay-as-you-earn tax

has a clause shifted to the position of a word.

Phrases can nest deep to many levels easier than other ranks. For example, a noun phrase embellished with adjectives and prepositional phrases:

the big thick book with the silky red cover on the bookshelf by the fireplace

has structure:

( the (big (thick book)) ( with ( the (silky (red cover)) ) ) )
on ( the bookshelf ( by ( the fireplace ) ) )

Finally, words consist of morphemes. Some morphemes are lexical, others are grammatical. For example, the word undiscerningly has structure:

( un
( discern //only one lexical morpheme
ing)
)
ly

Whereas compound word beekeeper has two lexical morphemes, bee and keeper. Morphemes are the atomic structure in English grammar.

Rank in Groovy
Like English, Groovy is best analyzed as having 5 ranks: (1) top-level, (2) statement, (3) expression, (4) path, (5) primary. It could be useful to match up the ranks of Groovy with those of English.

A top-level is a class, interface, or enum definition, standalone method definition or statement, or package or import statement. For example:

def mean(a, b){
def c= a + b
c / 2
}

It could correspond to a sentence in English. Class and method definitions consist of statements, just as sentences consist of clauses. A standalone statement can be a top-level, just as a simple sentence consists of only one clause.

A statement is of various types, e.g. if, while, try, break, expression, or block. A statement in Groovy could correspond to a clause in English.

A common type of the common expression statement is the assignment, e.g:

def b= c + ( d * e )

This could correspond to the subject-predicate style of clause, where b is the subject and the rest is the predicate.

A block statement could correspond to the complex portion within a compound-complex statement.

An expression consists of path structures, just as a phrase consists of words.

One such structure is the closure, which itself consists of statements, entities of a higher rank. This is rank-shifting in Groovy, e.g:

def c= {
  it= it * 2
  println it
  it * 3
}(7)

Compare this with the rank-shifted compound sentence inside the relative clause:

The mountain range, of which the tallest was there and its peak needed knocking off, loomed before them.

Expressions enable the deepest nesting, just as phrases do in English. For example:

a + ( b * ( -c – d ** e ) / (z= f1 – (f2 * f3)) % g ) +
( ( !h1 || h2 ) ? i : -j )

In general, the unary operators have highest precedence, then left-associative binary, then ternary, then right-associative binary.

Each path structure consists of a head followed by path elements, e.g. arguments, subscripts, closures, names after . or ?. or *. or .& or .@ For example:

callMet(d, e, 2.71, "abcdefg")[7].&fin()?.gun()
.g2{->}.g3(1, "b"){->}.@hun.&@iun*.jun[a,b,c]

Rank-shifting is far more likely within Groovy path structures, than in English words. Path structures are eventually comprised of primaries.

Primaries such as identifiers, operators, literals, numbers, and strings are the atomic structures of Groovy grammar, just as morphemes are the atomic structure of English. The operators ( + * % || && ) and literals (true, null, this) are like grammatical morphemes, while identifiers are like lexical morphemes.

Summary
The matching between Groovy and English syntactic rank isn't perfect, especially for the lower ranks, but does go a fair way. Groovy copies its 5-rank structure from other programming languages, such as Java and C++. I suspect the matching 5-rank structure makes it easier for English speakers to write and read programs in these languages. Perhaps people who don't already know Java or C++ would benefit from seeing comparisons with English grammar when they learn Groovy.

The Groovy Visions

2008-03-30T03:55:00.000-07:00

Closures for the JVM
James Strachan had a vision: he wondered why the Java platform couldn't have closures. So he began building Groovy, announcing it on 29 August 2003. He attracted others with a similar vision, such as Geir Magnusson and Richard Monson-Haefel, and together they lodged Groovy as a JSR on 16 March 2004. This attracted the attention of Sun Microsystems, as also did Microsoft putting closures into C#. So the long tentative process of putting closures into Java 7 began. The prototypes have been built, and we all await the final decision. Perhaps it will be announced at the upcoming JavaOne 2008 conference.

Groovy for Grails
Guillaume Laforge had a vision: he wondered why a Rails-like web framework couldn't be written in Java. It would run faster than the Ruby-based Rails, but would need a dynamic language to configure it. So he took control of Groovy from James, and along with fellow despots Graeme Rocher, Jochen Theodorou, and Jeremy Rayner, started building Grails and putting a MOP into Groovy. A year ago at JavaOne 2007, according to Guillaume, was the genesis of G2One, Inc, where investors Bay Partners and executive Alex Tkachman joined them in the vision, enabling them to work full-time on building and marketing Groovy and Grails.

If it weren't for Guillaume's vision, Groovy would have died after it fulfilled its usefulness in James' vision. However, because it's so closely tied into the vision for Grails, Groovy may again die when it's fulfilled its usefulness in Guillaume's vision. G2One, Inc is a vehicle to quickly realize capital gains on perceived future income, calculated from present income and its growth rate. Consulting and training in Grails can generate huge earnings quickly, as Ruby on Rails has done, while Groovy's present purpose is to make it as easy as possible for Java developers to use Grails. There's no financial incentive to make it do anything more, and the majority votes of G2One shareholders indirectly control the majority decisions of the Groovy despotry.

Bringing in Newbies
On 29 August 2003, the same day James Strachan announced Groovy, I flew from Hong Kong to Wuhan China to begin my very first teaching job, leaving behind a programming career. While here, I've also developed a vision for Groovy. I've wondered why Groovy can't appeal to non-Java and new programmers, as well as those already on the Java platform. It may not appeal to Python or Ruby programmers, but what about to Perl, PHP, or VB programmers? I've wondered what else Groovy needs to bring newbies to the Java platform, begun on extensive documentation for them, and attempted stuff like Unicode name-aliasing and Lisp-style self-referentiality.

However, my vision for Groovy is longer term than G2One Inc's horizon for profiting from Grails, and may even be at odds with it. I heard that at Groovy DevCon4 in October last year some Groovy developers wanted to rename the Groovy Language after version 1.5 to shake me off. I figure if they decided to do it, it'll be in time for JavaOne 2008 next month, so I've set up GroovyScript just in case. If the Groovy developers rename Groovy 1.6 to the Grails Language or whatever else they might call it, I'll bundle it in GroovyScript just to keep Groovy's groovy name alive. I'll fight harder to keep the Groovy brand alive than James Strachan did.

But I'm hoping the Groovy developers made the right decision last year at the DevCon.

GroovyScript is Born

2008-03-25T06:58:00.000-07:00

There's been another change of plans. The Grerl-Vy scripting language is now known as GroovyScript. And it has a different focus...

1. Linux
I'm more convinced than ever that Windows is a toy operating system, OK for beginners, perhaps. I learnt Groovy on Windows, but now I work with Groovy on Linux. Any serious use of Java is on Linux, so GroovyScript should run on Windows, but excel on Linux.

It could utilize some syntax from the Unix legacy apps (bash/sed/awk/perl). For example:
    println "ls".execute().text
could be abbreviated using backquotes, as:
    println `ls`

GroovyScript could also specialize in the shortest possible one-liners in the Linux terminal. Instead of:
    groovy -e 'println Math.PI'
we'd write:
    vy 'prtn π'

2. Licence
GroovyScript runs on top of Groovy, but doesn't bundle it. Legally, I can only include “Groovy” in GroovyScript's name if it doesn't actually bundle Groovy itself. (Though if some future version of Groovy is renamed, I could start bundling from that version.)

When I began on Grerl-Vy, my horizon was limited to Groovy, so I chose the Apache licence v2 simply because Groovy used it. But now I see how my work fits into the big picture, how GroovyScript sits on top of Linux and Java. And since they both use the GPL licence, so should GroovyScript for that reason alone.

But there's another issue at stake: as Richard Stallman recently said "There are many companies that are looking for a chance to make free software proprietary. The GNU GPL's job is to defend your freedom as a user, regardless of who might try to take it away.”

3. Lisp
Wanting a self-referential language syntax for GroovyScript, I started off using JParsec, a context-sensitive combinator parser, to escape the code tangling Antlr 2 requires to lex and parse a language as complex as Groovy. After seeing how lightweight Ken Barclay's Groovy-based parser was, I then tried using that instead.

However, I now doubt I can bring self-referentiality to GroovyScript with anything less than a Lisp-based parser. As I once read somewhere: all roads lead to Lisp. Groovy specializes in what's below the AST, while Lisp specializes in the syntactic level. So I'm now exploring the Java-based Kawa implementation of Scheme: does it perform properly and will it integrate with Groovy?

Although I've skimmed through Lisp tuturials before, I don't really understand it. Having some tangible project to use it for will help. And even if merging Scheme and Groovy proves impractical, I may still experience Eric's famous epithany of enlightenment!

Linux and Lisp

2008-03-20T03:25:00.000-07:00

A few months ago I got a slight shock while surfing the web. Following a link from Reddit, I took a look at this web site and scrolled down to the first picture. I remember jumping, thinking “How did a picture of me get there?” A few seconds later: “Oh, that can't be me. The guy's wearing a Linux T-shirt. I've never worn one of those.” I don't think I've ever seen a lookalike of myself before. There's a few slight differences, like the nose, but the T-shirt was the most noticable. And I'm sure with a nickname like “Maddog”, his personality's quite different to mine. But what's that he's saying? “GRRRRRRRRRRRRRRR...”? Finish the word: “GRRRRRROOOOOOVEEEEEEEY!”

It's been a few years since I've let the beard grow that long. The glasses are exactly the same as the ones I had until about 6 years ago. One difference others would notice is he tilts his head to his right, while I tilt mine to my left. However, to me there was no difference when I looked in the mirror. The reason he and I tilt our heads is to compensate for a wonky eye. When tilting my head one way, I see double vision, but not when I tilt it the other, so I grew up with the habit to force better vision. I had an eye operation 10 years ago which improved things a little, but old habits die hard. I've seen that picture in the mirror many times when my beard was longer and I had those glasses. It's hard to describe the jolt I had when I first thought someone had posted a 10-year-old picture of myself on an internet site dedicated to famous programmers.

At about the same time as that, I upgraded to Linux. It was taking Windows XP mere weeks to start exhibiting slave-machine behavior after a fresh install, there's too much bad rap written about Vista, and all my software was Java-based or open source. So using Wubi and LVPM/Lubi, I installed Ubuntu without any CD/Roms. And I found an English version of O'Reilly's Linux in a Nutshell up here. Now, after 3 months of using Ubuntu, I'm not sure if I could go back to Windows. Everything just works! Although I've had many encounters with Unix and Cygwin over the years, nothing was permanent. Because Ubuntu's so easy to install and use, I'd imagine there'll be a mass conversion from Windows to Linux on the desktop in the near future. I'm hardly an early adopter of things: I'm not a geek, just a nerd.

I would have switched to Linux earlier if I'd kept up a serious interest in computing, but I got diverted by other interests over the last 10 years. When I lived in Australia, I became interested in business. When I left mainframe programming to teach English here in China, I became interested in linguistics. After a few years reading up on topics such as Functional Grammars, Transformational Grammar, Discourse Analysis, etc, I returned to programming as a hobby, inspired by Groovy. I started to see programming language structure in terms of natural language structure, both where they matched and where they differed. Then I stumbled on something Larry Wall once wrote: Perl is a mess because English is a mess. Reading further, I found out Larry had studied natural language theory as well as computer languages at college. Perhaps understanding natural language structure brings a different perspective to analysing programming languages.

I learnt to use collections, closures, and builders in Groovy, typical dynamic language fare. Then I learnt about things that backtrack: regexes, a little Prolog, and combinator parsing. I had learnt to read many Chinese characters up here, and soon saw the potential to make program code much terser if I could use them everywhere in my programs. So I used Groovy for prototyping this, learning about Unicode. A year ago, I first announced Grerl: a scripting language inspired by Perl's tersity and the potential of Unicode for the Groovy ecosystem. But why does the Groovy Language need a scripting language?

The rise of Python seems to have created a new niche in programming languages. Programming languages have traditionally been divided into two groups: systems languages such as Java and C, and scripting languages such as Beanshell and Perl. A third niche has arisen between systems and scripting languages, into which dynamic languages like Python, Ruby, and Groovy have entered. These dynamic languages can scale better than scripting languages, but aren't adequate for performance-critical code. Perl was designed for scripting, for which it was great, but then it moved into the dynamic space of larger programs, for which it's less suited. Computer languages for larger systems require more stringent designs, just as English for essay-writing needs to be more formal than English for casual conversations. Its developers rejecting perlish features, Groovy fills the middle niche quite well, but doesn't fill the scripting niche as well.

Renaming Groovy?
Late last year, I heard from a reliable source that some Groovy developers wanted to give the next version of Groovy a different name because they saw me as a threat to their future profit from Groovy, but to keep the Groovy name for the present version so I couldn't capitalize on the vacant brand. So I announced Vy as a wild attempt to ensure the Groovy name would always tag the best, most recent, version of the language, whether it's controlled by the developers at Codehaus, or it's a composite package tagged as Gr+OO+Vy by me. Later, I combined the Grerl and Vy names together as Grerl-Vy.

Unfortunately, I raced to get a beta of Grerl-Vy 1.0 out the door in time for a rename of Groovy, which resulted in buggy and incomplete code. I won't do that again: there'll be no betas for Grerl-Vy 1.1. The first release will be RC-1. Neither will there be a release schedule: Grerl-Vy is as much about language design as implementation, and design can't be rushed. Version 1.1 will be a minimal, but fully functional, subset of Groovy: it includes the features Groovy added to Java, but excludes most of what Groovy inherited from Java. There'll be closures and special syntax for expandos, but no class or function definitions. There'll be lists and maps, but no arrays. Everything will be dynamically typed. Grerl-Vy 1.1 replaces JParsec with a custom combinator parsing library. It might look like a Groovy version of JavaScript: perhaps I'll call it GroovyScript. And is the Apache License the best?

In implementing Grerl-Vy, I'm betting on two unknowns: that Groovy 2 and the new MOP will be much faster than Groovy 1.5, and that there will be closures in Java 7. Some portions of Grerl-Vy's code, such as the parsing library, will be ported to Java 7 when it arrives, for which I'll need closures. The rest of Grerl-Vy's code, the user-configurable portion, will be ported to Grerl-Vy itself: it will be self-referential. I've taken a recent detour into Scheme and Lisp in order to fully understand self-referentiality. Lisp's macros are simple, dealing only with pairs and closures, while Grerl-Vy 1.1 self-reference must be more complex, dealing with lists, maps, GStrings, regexes, blocks, and closures. I hope I can make it happen.

I haven't heard any more about a name change for Groovy, and nothing's been mentioned online, but that doesn't mean it won't happen. If the Groovy developers have decided to rename Groovy, how would they do it? If they follow past form, they would wait until just before a major release, perhaps Groovy 1.6. A developer, but not a despot, would suggest the idea on the mailing list, then a few others would join in concurring. Guillaume Laforge would elicit feedback from users. When the loss of brand advantage was mentioned, a marketing blitz for the upcoming JavaONE conference in May 2008 would be suggested. Then the decision would be made to rename: everything would happen quickly. Perhaps Grails would also be renamed. Does G2One and G2X mean 2 G's to 1 ? What is X?

Self-Referentiality

2008-03-13T08:05:00.000-07:00

Paul Graham writes in Revenge of the Nerds:
“Lisp and Fortran were the trunks of two separate evolutionary trees. [...] These two trees have been converging ever since. Lisp started out powerful, and over the next twenty years got fast. So-called mainstream languages started out fast, and over the next forty years gradually got more powerful, until now the most advanced of them are fairly close to Lisp. Close, but they are still missing a few things.”

Paul lists 7 Lisp features that mainstream languages have acquired: conditionals, function types, recursion, dynamic typing, garbage collection, expression-based syntax, symbol types. He also mentions 2 Lisp features not yet part of mainstream languages: a code notation using trees of symbols and constants, and the whole language always available.

Having the whole language always available means there is no distinction between read-time, compile-time, and run-time. We can compile or run code while reading, read or run code while compiling, and read or compile code at runtime. This is the basis of self-referentiality in Lisp programs, making their syntax more powerful.

Natural languages are self-referential. For example, a common English-language idiom when listing the similarities of things but before listing their differences is to say “And that is where the similarity ends.” The “that ends” refers to the physical text being read, or speech being spoken. The content of the message is refering to its own medium.

Self-Reference in Grerl-Vy
Grerl-Vy is a programming language I'm designing and implementing. Its name is a composite representing two visions. The Grerl vision is to write code with even greater tersity than present terse languages, using more syntactic shortcuts and more Unicode tokens, enabling these via self-referentiality. The Vy (pronouned “vee”) vision is to stay close to the groovily-named Groovy, using it both as the implementation language and as the target AST, and being ready to ensure the Groovy brand's continued association with the Groovy technology if necessary.

While designing Grerl-Vy, I'm trying to simplify the complex Groovy AST nodes with a more minimal syntax, in the hope I can make it elegantly self-referential. However, it must still be easily readable to programmers used to the C++/Java/Groovy syntax. In my experimenting, I'm beginning to understand what the saying “Whoever doesn't understand Lisp is doomed to re-invent it” means. It only takes a few minutes to alias method names in Scheme: in the Java-based Kawa, we can alias car with 头 and cdr with 巴, then write more readable programs using them. But I'm restricted to Groovy because of the Vy vision.

All statements will have expression syntax in Grerl-Vy to make it easier to write macros. To help with this, I'm introducing a new syntactic shortcut: just as Groovy allows a closure as last parameter to a subroutine call (i.e. function, method, or closure call) to be written after the other parameters outside the parentheses, Grerl-Vy will also allow a map as last parameter (before the closure) to be. Because Grerl-Vy decides during parsing whether to implement a syntactic subroutine call as a MethodCallExpression node or SwitchStatement node or whatever in the AST, we can write statement syntax more elegantly.

For example, the switch statement would have syntax:
  switch(value)[
    ... : { statements },
    ... : { statements },
  ]{ statements }

This looks like a subroutine call in the syntax, but the parser would intercept the “switch” keyword and implement it as a SwitchStatementNode in the AST. (The return, break, and continue keywords won't be available in Grerl-Vy release 1.) There will be a placeholder method head in the vy.lang.System class:
  void switch(
    Object value,
    LinkedHashMap cases,
    Closure default= null
  )

By reducing the syntactic complexity of Groovy in Grerl-Vy, we can introduce self-referentiality more easily.

Vy will put the V in Groovy.

JRuby and Jython: Groovy's peers

2008-03-10T07:58:00.000-07:00

Sun Microsystems recently hired Frank Wierzbicki and Ted Leung to work on Jython full-time at Sun. Glen Smith recently blogged concerning this and his meeting with James Gosling of Sun in Australia:
“One question that was asked was about the recent hires around JRuby & Jython, is Sun sidelining Groovy? James had some nice things to say about Groovy, actually, and reiterated that really it's a numbers game for Sun. Big numbers of Ruby and Python developers to lure to the platform, fewer (but growing) numbers of Groovy developers already productive there.”

In Charles Nutter's blog entry on the same subject he writes:
“So what does this mean for Sun? Well, it means we're serious about improving support for Python on Sun platforms. Jython is a big part of the story, since they have many challenges similar to JRuby, but a bunch of new ones as well. So we'll be looking to share key libraries and subsystems as much as possible, and we'll start looking at Jython as another driver for future JVM and platform improvement.”

As suggested in a recent blog entry of mine, the JVM needs to keep up with Microsoft's DLR support of IronPython and IronRuby. By adding the Jython experts to the JRuby ones already on the payroll, Sun is better able to build a solid MLVM to compete. Groovy must consider whether to also program to such a MLVM, about which Alex Tkachman says: “I think it is a serious effort for Groovy but i am for 100% sure it is doable. There is nothing I am aware about that prevent that in theory.”

Now Charles Nutter has blogged about Duby, JRuby's static-mode answer to what some Groovy developers would like for Groovy but others don't.

If Sun builds a MLVM with a tree-like DLR-style interface, and re-engineers JRuby and Jython for it, then Groovy must follow suit to stay relevant. But Groovy doesn't need to build a static Doovy mode because Java is Groovy's Duby.

Defining String Syntax, part 2

2008-02-28T06:44:00.000-08:00

Making extensive use of comments and strings can be quite a chore when our programming language's delimiters, escapes, and what not clash with what we want to embed. Continuing from my earlier entry on comment and string syntax, let's look at possibilities for programmer-defined comment and string processing in Grerl-Vy...

Comments
Comments are defined by their opening and closing delimiters. In Groovy and Java these are /* and */, plus // and newline. A programming language could allow programmers to configure comment syntax using meta-syntax. If we used the object-oriented coding paradigm for this, we'd expose a configuration object, say CommentSyntax, with the method:
CommentSyntax.addStyle(
  String name,
  String openingDelimiter,
  String closingDelimiter,
  boolean isSelfNesting, //do we allow nested comments with same syntax?
  List otherNestedStyles = [], //optional parameter
  char escapeClosingDelimiter = null //optional parameter
)
The comment could allow other comment styles with the same closing delimiter but different opening delimiters to be nested. The comment could also allow the closing delimiter to be ignored if escaped by another character.

So Groovy and Java comments could be defined by:
CommentSyntax.addStyle("blockComment", "/*", "*/", false)
CommentSyntax.addStyle("lineComment", "//", "\n", false) //ignore OS variation

Comments are sometimes used in programming languages as mini-languages parsed separately, e.g. Javadoc, but a more correct way to do this is probably to use annotations.

Strings
Specifying string syntax is more complex than comment syntax because we're now concerned with evaluating its value: specifying nesting rules is not enough.

As with comments, we need to know both the opening delimiter, e.g., " or ' or """, and the closing delimiter, which is often the same as the opening one, but needn't be. Some string syntax, such as single-quoted strings in Bash, doesn't allow its own delimiter to be part of the string – not a very nice option. To allow embedding of the closing delimiter, we need an escape character, usually \. In Pascal, we embed the closing delimiter by writing it twice, though this is the same as the closing delimiter being its own escape character.

When the closing delimiter and its escape character are different, we can't embed the escape as the very last character in the string. We can allow the escape to escape itself when it's the last character, e.g.:
  'Woody\'s wish is to end strings with one backslash (\) like so... \\'
would print
  Woody's wish is to end strings with one backslash (\) like so... \

We could therefore define a basic string syntax using:
StringSyntax.addStyle(
  String name,
  String openingDelimiter,
  String closingDelimiter,
  char escapeClosingDelimiter = null, //optional parameter
  boolean escapeEscapeAtEnd = false
)

The simplest string syntax in Groovy is the special-purpose regex string syntax:
  StringSyntax.addStyle("regexString", "/", "/", "\", false)
Groovy regex strings can't embed the escape character as last character (though because they're actually interpolated, the workaround is to put a null interpolation afterwards).

Multiline strings in Groovy and many other languages let us use the close-delimiter-escape character as a line-continuation character when it's the last character before a newline. Java, Groovy, and others also allow us to use it to generate control and other non-keyboard characters, e.g., \n for newline, \a for alert, \x20AC for euro symbol, etc. Because these languages make such wide use of the escape character within strings, they therefore require us to escape the escape everywhere in a string, not just as the last character.

We could specify all this with an extra Map-typed parameter in the StringSyntax.addStyle() method, e.g.:
  [ "\": "\",
    "n": 0x000A as char,
    /x([0-9a-fA-F]{4})/: "$1" as char,
    ...
  ]

Strings are quite often used to embed little languages in a more general-purpose programming language. Groovy's regex strings were a special syntax added because the quoted language itself also used backslashes as an escape. By letting programmers define their own string syntax meta-syntactically, they can similarly cater for new little languages they might need to embed as strings in Grerl-Vy in the future.

Finally, we need to specify whether interpolation is enabled, and what the escape character is. There could be more than one. If there's any interpolation escapes different to the standard escape, as in Groovy where $ means interpolate, we must be able to standard-escape that interpolate-escape.

We'll specify comment and string styles using special calls within the source code, either at the top of a file or applying to a curly-delimited block of code. By giving such meta-syntactic constructs the same syntax as the Grerl-Vy language being parsed, perhaps in future we'll be able to make Grerl-Vy's syntax self-referential.

Grerl-Vy is coming.

Grerl-Vy Take 2

2008-02-14T22:24:00.000-08:00

For my first attempt to create my own language on top of Groovy's AST, I made some mistakes: trying to use every AST node and option before releasing a fully working prototype, using the API-heavy non-Groovy JParsec, and designing the language as I was coding it. But, hey, it was still useful to do: I now know a lot more about both the Groovy AST and combinator parsing as a result, and better understand what's possible for Grerl-Vy.

I'm building Grerl-Vy again from scratch. (You didn't think I'd want to write tutorials again, did you?) But I'll build it up carefully step by step this time, aiming for depth in each iteration rather than breadth, so each release is a fully working useful language. Instead of JParsec, I'll write my own combinator parsing library in Groovy, using Ken Barclay's GParsec as a starting point. And I'll have a simple design in mind before I start.

The Design
The first fully working version of Grerl-Vy will use lists and maps only, no generics or arrays. Objects only, no primitives. Expressions only, no statements or top-levels. Any structure using a statement AST node will be embedded within a called closure which returns some value so it'll look like an expression. The break, continue, and return keywords won't be available, so blocks and closures will behave similarly anyway. Grerl-Vy will promote Lisp-style coding!

For example, the for-statement presently has smelly syntax:
  for( ... in ... ){ ... } //present syntax
  for( ... in ... ) ... //alternative present syntax

The new syntax will be:
  for( ..., ... ){ ... } //new syntax looks like a method call
  for ..., ..., { ... } //alternative new syntax for single line call
  c = for( ..., ... ){ ... } //new syntax can return a value

A closure will be able to stand alone in the syntax, without needing a label (which aren't available anyway) or -> to distinguish it. A block will require a try statement, so it looks like a method call:
  try{ ... } //looks like a method call with a closure param
  c = try{ ... } //can also return a value

Because we want to make use of the rich capabilities of Groovy's switch statement, we'll also make that look like an expression:
  c = switch( ..., [ //looks like method with LinkedHashMap param
    ... : { ... }, //fall-thru not allowed
    ... : { ...; ... } //last value of executed case returned
    $t||$f: { ... } //default, ie, $t means true, $f means false
  ])

We'll put more operators into Grerl-Vy. Ecmascript 4 specs the &&= and ||= operators. We're keen on Python's magic _ variable. An operator called /% could return a list with both integral dividend and remainder. We could re-introduce === for strict equals, along with !== from JavaScript. We could negate the matcher with !~ as in JRuby, and the finder with !=~. If we use :: for Groovy's in, we could use !: for not in. Here's some excellent ideas from The Lurker. Just as ++ and – – have pre- and post-increment forms, so should other operators. For example, (i = j := 3) could mean assign 3 to j after assigning j to i. We could swap the values in variables by writing (a = b := a). The Lurker also suggests assignment shortcuts could have pre-action shortcuts also, eg, (i +:= 3).

Once we've got a fully-working expression-based language out, we'll play with it to see where it needs improving.

Grerl-Vy is still alive.

Dynamic Language Runtime

2008-02-12T02:20:00.000-08:00

Recently I discovered that Microsoft's Dynamic Language Runtime (DLR) presents a tree-like interface to programmers. Both Jim Hugunin and Martin Maly of Microsoft have blogged about this. The alpha edition of IronPython 2.0 is slated as “the first release of IronPython to target the DLR”. IronRuby is also targeting the DLR, while the earlier Ruby.NET targeting the CLR directly has recently been shelved. Other languages from Microsoft to target the DLR are VB and JScript.

The JVM seems to be a little behind Microsoft's CLR in this regard. Shortly after DLR's announcement last May, Charles Nutter of Sun created a Java Language Runtime repository for shared use code, but only one small piece of code has been put there. More recently, John Rose has started the Da Vinci Machine project. However, Charles writes that he's not sure whether Sun will follow through with a multi-language VM.

There's some of us who'll never program to Microsoft technology again, being fed up with the way they stop improving their products as soon as they get their market share. But the temptation to develop to the DLR on Mono under Linux could prove too great if Sun stops innovating for the JVM. However, because I like Groovy and have some recent experience programming to its AST nodes, I'm thinking maybe Groovy's AST could become a popular tree-like interface to the JVM. And now that Grails 1.0 is out, Groovy needs another avenue to extend its reach to the programmer community. Of course the Groovy AST is at too high a level of abstraction to be equivalent to the DLR, but maybe in future the Groovy developers will enable more, lower-level, nodes on their AST.

Here's some code that parses a Groovy-like grammar and writes directly to the Groovy AST. (Those who don't know JParsec may find it hard to read, and of course it needs some improvement.) Because Groovy's AST might make an excellent DLR-tree equivalent, I've decided to suspend my work on Grerl-Vy and begin writing some tutorials to show programmers how to similarly create their own programming language to write directly to the Groovy AST. If I, with my business programming background, have wanted to write my own programming language syntax, then I'm sure many other programmers will too.

Although Jochen Theodorou's written that Groovy 1.6 will have AST node changes, I'll use Groovy 1.5 for the first drafts, along with JParsec for the lexing and parsing, as I've learnt it and, like Groovy, it's a Codehaus product. However, Groovy book author Ken Barclay has suggested such parsers can be written directly in Groovy easily, so I'll probably investigate that further later on.

Goodbye Grerl-Vy for now, and welcome "Groovy DLR".

Grerl-Vy's (and Groovy's) Symbols - part 2

2008-02-06T22:17:00.000-08:00

Parsing technology has advanced rapidly over the last decade or two, so now programmers can easily define languages using recursive-descent definitions, which perform almost as well as hand-coded bottom-up parsers. Java's Antlr and Haskell's Parsec spring to mind. It will become increasingly common for programmers to use their own custom languages to write code, just as they presently use their own IDE settings and code formatting, and even choose their own IDE's. Program code they write will be generated from its AST-form into a standard syntax and format before being shared.

Grerl-Vy is a custom programming language I'm writing, for myself to use, that writes to the Groovy API. I intend it to be much terser than Groovy, so all grammar is represented by symbols, such symbols optionally overloadable with names from any natural language, and the JDK/GDK English-language names optionally aliasable by names in any other natural language. When I'm finally using this language myself, I'll release it as open source so any others interested can copy and modify it to define their own custom languages for the Groovy API also.

I've been looking at the syntax of other programming languages to get some ideas on what extra symbols I can put in Grerl-Vy. I've previously blogged about Grerl-Vy's symbolic replacements for keywords in Groovy, with some very tentative examples. Let's look now at other possibilities, first considering the existing symbols in Groovy. (I've ignored symbols with alphanumerics here, such as \t or 0xFF, as there'd be too many to list.)

Java has 50 such symbols:
\ various escapes
/* */ comment
// comment until end of line
; separate statements, etc
" " quoted string
' ' character (Java); quoted string (Groovy)
.* all classes in package, or methods in class
- unary minus, binary minus
++ in-place increment, both prefix and postfix
-- in-place decrement, both prefix and postfix
% modulus
/ divide
* multiply
& binary and
^ xor
| binary or
! logical not
~ bitwise negate
!= not equals
>>> big right shift
>> right shift
<< left shift
== equals
= assign
-= assign with minus
+= assign with plus
&= assign with logical and
>>>= assign with big right shift
>>= assign with right shift
<<= assign with left shift
%= assign with mod
/= assign with divide
*= assign with multiply
|= assign with logical or
^= assign with xor
( ) method or closure call, etc
[ ] subscript, in-place list or map def, etc
{ } enclose statements block, closure def, etc
? : conditional expression
: labels; separate map key and value
@ annotation
. path to next element; decimal point
... variable args
+ unary plus, binary plus
> greater than, close generic brackets
<= less than or equal to
< less than, open generic brackets
>= greater than or equal to
&& logical and
|| logical or

Groovy, its developers wanting to avoid "perlishness", only adds another 20:
[:] empty map
=~ regex find
==~ regex match
<=> compare
** power
**= assign with power
.. exclusive range
..< inclusive range
.@ path to field, never to property
-> closure parameters
?: Elvis operator
?. null-safe path to next element
*. spread path to next element
.& path to method pointer
#! comment at beginning
$ interpolate string
/ / quoted string
""" """ quoted multi-line string
''' ''' quoted multi-line string
, separate list elements, etc

Java 1.4, wanting to keep up with other languages, added the standard syntax of regexes. Because they're such an integral component of Groovy, they should be considered part of Groovy's syntax. Regex uses 30 symbols, though many are also used in Groovy proper:
. match any character
{ } match exact number of times
{ , } match in range of times
[ ] character class
( ) capturing group
\ various escapes; quote following character
^ character class not; match beginning of line/input
$ match end of line/input
| alternation
? greedy option
* greedy repetition (zero or more)
+ greedy repetition (one or more)
(? ) set flag/s
(?- ) unset flag/s
# comment
- character class range
&& character class intersection
(?: ) non-capturing group
(?> ) atomic group
?? lazy option
*? lazy repetition (zero or more)
+? lazy repetition (one or more)
{ , }? lazy match in range of times
?+ possessive option
*+ possessive repetition (zero or more)
++ possessive repetition (one or more)
{ , }+ possessive match in range of times
(?= ) lookahead
(?! ) negative lookahead
(?<= ) lookbehind
(?<! ) negative lookbehind

Remember I'm not counting the huge number of mixed symbol/alphanumeric combinations, such as \Z and (?x).

Formatter syntax (aka printf), added in Java 5, gives many more symbols, though most are mixed with alphanumerics:
% substitute value
% $ substitute specified value
%< substitute previous value
There's also a Java internationalization API defined in java.lang.text, with its own little formatting language, though I suspect most Java/Groovy programmers prefer to use the Formatter/printf syntax, being familiar with it already from Unix and C.

There's other miscellaneous places where symbolic syntax is added to Groovy:
/** */ Javadoc comments
@ various extra info in Javadoc
$ match reference in regex replacement string

Some other JVM languages considered to various degrees to be Groovy's peers might have symbols I could use in Grerl-Vy. The syntax of JavaScript, embedded in Java 6, either copies Java's or is copied by Groovy. The only unique symbols I could find:
=== strict equals
!== strict not equals

Scala, although statically typed, does have inferred typing and is much terser than Java. Most of its symbols are names within its class libraries. The only syntactic symbols are:
<: upper bound generic type
>: lower bound generic type
<% view upper bound generic type
# static path selection
=> call by name; etc
@ bind to following pattern
<- for-comprehensions
_ wildcard pattern

The <: and >: symbols might be nice Grerl-Vy replacements for Groovy's extends and super keywords within generics.

JRuby has some more symbols:
# comment to end of line
? character
?\ - control and/or meta character
\ various escapes
' ' string without interpolation
" " string with interpolation
` ` string with echoed interpolation
% string/array/regex follows
<< here-doc with interpolation
<<' here-doc without interpolation
<<" here-doc with interpolation
<<- here-doc with indented end-marker
: symbol
:' ' symbol without interpolation
:" " symbol with interpolation
.. inclusive range
... exclusive range
[ , ] arrays
{ => , } hashes
$global global variable
@@class class name
@instance instance variable
$ various predefined globals
$x various predefined globals, for punctuation x
__XXXX__ various predefined variables
:: class::member
!~ similar symbols to Groovy, but also !~
[< ] superclass in class defn

Much programming language syntax has already been catalogued elsewhere. I'm comparing symbols in different programming languages to see which are the generally accepted ones. I believe programming language symbols will merge towards a single standard, just as math symbols have over the centuries. The present C++/Java symbols are in that standard, being the most popular languages and having influenced others such as Groovy and PHP. I only want recognized symbols in Grerl-Vy.