For quite some time now there have been appearances of a new type of
declarative modules on CPAN. Some of these (but not all) are
MooseX::Method::Signatures, giving you a method
keyword with signatures and type validation; TryCatch
which provides you with a modern and elegant way to catch exceptions; and
MooseX::Declare, a whole extendable, declarative framework around the
Moose ecosystem.
To many people who don’t have the time or the interest to live on the edge of advancements in the Perl community, this might seem like magic trickery, much like source filters. And because of this many people keep a safe distance from all these developments, like they rightfully do with source filters. You should never use something before you know how it works on a deep enough level for decision making. This is why I decided to write this post, and to show how all of this is made possible.
The heavy lifter with all these syntax extensions is Devel::Declare, which was thought up and implemented first and largely by the mind of Matt S Trout, core team member of Catalyst and inventor of DBIx::Class It allows Perl modules to install hooks for barewords that will be executed when perl encounters the token in the code. There are two things that are important to notice here:
The hooks are defined in Perl space. That means you use Perl to extend Perl. There is no visible XS involved in developing these extensions. I have no idea whatsoever about C, C++ or XS. Paint me happy if it’s Scheme, Perl or JavaScript, but otherwise I wouldn’t trust me further than I can throw myself. But even though I have no experience in any of these low-level languages, I am able to contribute regularly to
MooseX::Declare, and build a (yet unreleased) extension of it called CatalystX::Declarative. This is a huge advantage since it means a lot more people can contribute, extend, document, or fix bugs. Including me, of course.The hooks will only fire on barewords. This means that the handling code does not have to parse Perl to find the invocations of its syntax. The keywords can for example be safely used in strings. The parsing of the special syntax only starts at the place where the syntax is used. This is a huge advantage over source filters, who always have to deal with the file as a whole, and not with individual invocations. This small scope in which these extensions act also allows safe combination of multiple extensions at once.
MooseX::Declarefor instance reusesMooseX::Method::Signaturesfor methods and modifiers.
After being invoked, these extensions use Devel::Declare to walk the
statement from the invocation, stripping and interpreting parts as they walk along.
When all is done, they will inject a bit of generated Perl code where the invocation occured.
Using methods as an example, this
method foo ($x) {
# method body
}
will be roughly translated to this
method {
# autogenerated code (lexicals, validation, etc.)
# method body
};
With this approach, it never actually has to parse the body. How the trick with
the injected semicolon after the block works took me a while to understand. It
simply injects a call to B::Hooks::EndOfScope that will fire when
the block was compiled. After that happens it picks up again and inserts the
semicolon. Here comes some internal magic into the game, because the
extension will use Devel::Declare to give perl a code reference
in this place to use, and this registration handler will get the actual code
body. This is not as complicated as it sounds, but it allows certain meta data
to be accessible when the method is registered. This is especially useful
with Moose’ usage of the MOP.
The rest is all history. The block is passed as a code reference as usual in Perl and installed under a proper name in the calling class.
Devel::Declare provides lots of utilities for parsing tokens and
handling the injection of code, and it is still young and in development. While
the above might seem amazing enough, this is only where it starts.
Devel::Declare has the potential to hook into more than just
constant keywords, and people are already thinking about ways to make the parsing
of syntaxes declarative by itself. This might result in something similar to what
Perl 6 provides with grammars and rules, except in Perl 5.
I hope this article will help some of you to overcome their fears from these new, shiny but all the while very useful extensions, or even draw the one or the other to start contributing to one of the existing modules or the whole movement itself.
Excellent post! very informative
It seems quite telling that "This means that the handling code does not have to parse Perl" must be followed by "to find the invocations of its syntax".
It seems quite humorous for notes of the evils of source filters having to parse Perl to be followed by "Devel::Declare provides lots of utilities for parsing tokens".
You've managed to convince me to stay well away from modules that use Devel::Declare.
http://cpanratings.perl.org/dist/TryCatch notes "Make sure that you have the latest Devel::Declare or you can have some weird issues that won't seem to make any sense". This is the type of error that I expect from source filters and almost never see otherwise.
It is the bane and hallmark of ETOOMUCHMAGIC.
I'll grant that the trick for finding the starting bareword sounds less evil than a typical source filter. The ETOOMUCHMAGIC hallmark above makes me concerned that it may be still too magical to be relied upon.
The reason I went to look at TryCatch in the first place was because the "method" example above appeared to pile on the "magic". I hoped to find a module that used Devel::Declare in a way that didn't require complex magic in order to find the /end/ of the "declaration" (which might then be a more reasonable use of Devel::Declare).
But I stopped after reading the reviews.
Note that I wouldn't be shy about using a source filter if the syntax implemented by the filter was very orthogonal to Perl syntax (much like POD is). Those seem to be rather rare, however (some "comment" modules may qualify). It seems that most extension authors want to deeply merge their preferred syntax into Perl's syntax.
Similarly, if I find the time to study the magic of Devel::Declare and determine that it isn't ETOOMUCHMAGIC, then I wouldn't be shy about using an extension based upon it if that extension also chose to eschew piling on more magic.
I don't need even more cases of huge time sinks trying to figure out which piece of magic conflicted with what other piece of (perhaps quite mundane) magic that lead to some bizarre error deep in some guts.
But the "method" example above shows the typical attempt at deep merging of new syntax into Perl's syntax. I'd frankly rather have a source filter that did something that was /obviously/ not Perl code. I hate trying to read code where the syntax has been subtly redefined (beyond the inevitable problems that result from the non-subtle magic required to make things appear subtle) and so the Perl parser in my head can't parse it (because it isn't Perl but it looks like it might be).
tye,
> It seems quite telling that "This means that the handling code does not have to parse Perl" must be followed by "to find the invocations of its syntax"._
> It seems quite humorous for notes of the evils of source filters having to parse Perl to be followed by "Devel::Declare provides lots of utilities for parsing tokens".
"It doesn't have to parse Perl" does not have to be followed by "to find the invocations of its syntax." This is just the most significant difference to me.
And yes, D:D indeed helps you to parse tokens. Because despite the fact that you are no longer parsing Perl, you might want to handle things like barewords (like 'method'), identifiers (like '$foo') and strings. This is all very low-level. You can hook PPI in as well if you want.
> It is the bane and hallmark of ETOOMUCHMAGIC
Actually, all of this is the bane and hallmark of EXPERIMENTAL. We're a small group of people. There's lots of possibilities for development, but there needs to be a community around it to grow.
> The reason I went to look at TryCatch in the first place was because the "method" example above appeared to pile on the "magic". I hoped to find a module that used Devel::Declare in a way that didn't require complex magic in order to find the /end/ of the "declaration" (which might then be a more reasonable use of Devel::Declare).
Maybe there is a confusion about this: blocks like in foo { block } don't need to be parsed. They are simply skipped over. What we're mostly working on isn't getting it to work when all is correct. It is getting decent error messages out in all cases possible when there are errors in the syntax.
> I don't need even more cases of huge time sinks trying to figure out which piece of magic conflicted with what other piece of (perhaps quite mundane) magic that lead to some bizarre error deep in some guts.
Then you'll probably not be part of the community driving it forward. That's unfortunate, but the same can be said about any complex project until it reached its stable point.
> I'd frankly rather have a source filter that did something that was /obviously/ not Perl code. I hate trying to read code where the syntax has been subtly redefined (beyond the inevitable problems that result from the non-subtle magic required to make things appear subtle) and so the Perl parser in my head can't parse it (because it isn't Perl but it looks like it might be).
Well, I personally don't like mixed systems. I like the system to have an API that is integrated as nicely as possible in the most common use-cases. I also like them to be semantically obvious, which I often find hard to achieve with separate systems.
With regard to the "inner Perl parser," I know what you're talking about. I guess I can't say more than "one gets used to it." For me it's all just API. Maybe I'm thinking too lisp-influenced.
Sorry, missed hitting the "Reply" button for the above comment.