Moo and Type::Tiny

December 14, 2014

At the start of 2014 I was working on a project to further improve the LCFG client. When I hit problems with Moose and its memory usage I discovered the excellent Moo framework which provides all the nice bits but is much less heavyweight. As part of the Perl Advent Calendar for 2014 someone has written a great introductory article on using Moo along with Type::Tiny. I’ve learnt a few things, I particularly like the idea of a “type library” as a good way to organize all the local types.


Moose role hackery

February 8, 2011

For quite a while now I have wanted to have the ability to apply a Moose role without actually loading any of the associated code. I’ve finally come up with a solution which seems to do exactly what I need.

For a bit of background, Moose roles are what is often referred to in object-oriented programming as “interfaces”. They are used to declare that a class is guaranteed to implement a particular behaviour (attributes and/or methods) as part of its API.

A commonly used role which is available on CPAN is MooseX::ConfigFromFile which is used to declare that a class has the new_with_config and get_config_from_file methods. These are used to set the values of attributes from a configuration file. This works well in conjunction with other roles, such as MooseX::Getopt, which can detect when the ConfigFromFile role is implemented and load the values for attributes from either the command-line or a specified configuration file.

The problem is that the MooseX::ConfigFromFile code is a little bit peculiar and has a dependency on MooseX::Types::Path::Class (and thus MooseX::Types and Path::Class amongst others) which are not usually essential and lead to memory bloat for no good reason.

So, here is my solution, add these two lines:

my $extra_role = Moose::Meta::Role->initialize('MooseX::ConfigFromFile');
__PACKAGE__->meta->add_role($extra_role);

I can use this to state that my own configuration loader module does everything that the MooseX::ConfigFromFile role requires but I do not need to load (or even have installed) the MooseX::ConfigFromFile module file itself. This seems to work equally well when applied to a role or a class.


List::MoreUtils unexpected behaviour

February 16, 2010

I’ve long been a fan of the List::MoreUtils perl module so I don’t know why this “feature” has never bitten me before. The module provides a number of functions for manipulating lists, in particular I was using any() and none().

any() returns a true value if any item in LIST meets the criterion given through BLOCK, for example:

 print "At least one value undefined"
               if any { !defined($_) } @list;

none() is logically the negation of any. It returns a true value if no item in LIST meets the criterion given through BLOCK, for example:

 print "No value defined"
               if none { defined($_) } @list;

The gotcha here is that both of them will return an undef value when the list is empty. It’s not such an issue with any() but this particularly caught me out when using none() as I was expecting it to return true when the list was empty. To my mind it really should return true as an empty list definitely doesn’t contain any elements which match the condition. Surely other people have had the same experience. In future I think I will stick to using the standard grep() and just get the size of the list returned.


UKUUG Advanced Perl Workshop

March 2, 2009

Last Thursday I was in London to attend an "Advanced Perl Techniques" workshop organised by the UKUUG. The tutor was Dave Cross, who has written a couple of Perl books. He has a good style of delivery, he was generally very knowledgeable, the presentation was well structured and amazingly it all ran to time (that takes real talent). Given the title and the list of topics I had high hopes of learning some really cool new things to do with Perl. Here’s the list of subjects which were covered:

  • What’s new in Perl 5.10
  • Dates and times
  • Testing (including coverage analysis)
  • Database access with DBIx::Class
  • Profiling
  • Object oriented programming with Moose
  • Templates
  • Web application development with Catalyst

Specifically, I wanted to learn more about DBIx::Class and Catalyst and find out whether I am using Moose in the right/expected way. I guess, looking at that list again now, I should have realised that it is a lot to get through in one day and necessarily it was only going to be a shallow coverage of each topic. Other than the Catalyst stuff at the end I thought it was all pretty good (if lacking in the really deep detail I wanted) and I did get some useful bits and pieces from the day. I felt the Catalyst section was done very lazily though, it had the feeling of being added as an after-thought and I wondered if it was actually just copied from the standard Catalyst documentation.

I was interested to learn that "DateTime" is considered the "best" module to be using for all time and date manipulation. It certainly has capabilities way beyond that which I was previously aware. I also found the profiling section interesting, I will definitely be looking at "Devel::NYTProf" in more detail sometime soon. The "What’s new in Perl 5.10" section was also particularly good and has encouraged me to start looking at the new features in more detail and, at least, start using them in any scripts I write for personal use. It’s a shame we won’t see 5.10 in RHEL5 but that’s the price we pay for system stability. By the time we get RHEL6 it will at least have had any bugs found and fixed by users of Fedora, Debian and Ubuntu.

All in all, it was worth going to the workshop. At some point in the future I’d love to see a "Really Advanced Perl" workshop which really goes beyond the beginners guide for DBIx::Class, Moose and Catalyst and demonstrates some of the more complex possibilities.


Running commands from perl

February 13, 2009

I think most people are aware of the dangers of running external commands from Perl, particularly when they involve variables storing user input. Over the years multiple methods have been provided to make things safer, such as system() taking a list of command parts and not using the shell rather than the original single string-based approach which forks a shell. That’s fine when you just want to fire off a command and check the return code, often though we want to capture the output for immediate perusal or store it directly into a file. Here’s a snippet of code from the subversion component I used when I wanted to run the svnadmin command and send the output direct to a file:

        my $out = IO::File->new( $outfile, 'w' ) 
              or $self->Fail("Could not open $outfile: $!");

        open( my $in, '-|', $svnadmin, 'dump', $repo_path )
            or $self->Fail("Failed to dump svn repository $repo_path: $!");

        while ( defined ( my $line =  ) ) {
            print {$out} $line;
        }

        $out->close or $self->Fail("Failed to save svn repository dump: $!");

All that just to do the equivalent of "svnadmin dump > /path/to/dumpfile". It could be even more complicated if you got an error code returned and you wanted to examine the output on stderr.

There are various solutions to this problem based around IPC::Open2 and IPC::Open3 but they can still be quite hairy and involve opening and reopening file handles.

I have recently discovered a much nicer, simpler interface which hides the complexity in solving these problems, the module is named IPC::Run. A quick example from the module summary shows the potential:

   ## First,a command to run:
      my @cat = qw( cat );

   ## Using run() instead of system():
      use IPC::Run qw( run timeout );

      run \@cmd, \$in, \$out, \$err, timeout( 10 ) or die "cat: $?"

      # Can do I/O to sub refs and filenames, too:
      run \@cmd, '<', "in.txt", \&out, \&err or die "cat: $?"
      run \@cat, '>', "out.txt", '2>>', "err.txt";


      # Redirecting using psuedo-terminals instad of pipes.
      run \@cat, '<ptypty>', \$out_and_err;

This is barely scratching the surface, it can act like Expect and do loads of other clever tricks.


Fuzzy Testing

October 13, 2008

I’ve been thinking about various aspects of the upcoming project to create a testing framework for LCFG components. One thing we really need is a replacement for what I am going to term “fuzzy matching”. With the current framework there is support for embedding tags [% and %] to mark sections which are going to change with every run (timestamps, version numbers, etc.)

This is a really nice idea and is very useful but there are a couple of downsides to the current implementation. In particular, the tags need to appear in the expected output and also the generated output (and consequently the code which is being tested). However, we do not want these tags to be appear in the generated output on a “live” system so we end up having to rebuild the packages in a special way to get the tags inserted into the code producing the output. This leads to a situation where the code we are testing is not identical to that on the live system, at best the difference is a few strings but in some cases completely different code paths are followed.

Having thought about this for a while I reckoned it should be possible to do away completely with the need to add tags to the generated output and just markup the expected output. Effectively the expected output becomes like a template. The inspiration for this approach came from Template::Extract and initially I thought I could just build directly on top of that module but I didn’t have too much success.

I’ve now come up with Test::FuzzyMatch, this is most definitely alpha-quality software but it is already quite useful. I think this demonstrates what I want to be able to do:

Here is part of a logfile from the boot component:

06/10/08 03:02:01: >> run
07/10/08 03:02:02: >> run
08/10/08 03:02:01: >> run
09/10/08 03:02:01: >> run
10/10/08 03:02:01: >> run
11/10/08 03:02:01: >> run
12/10/08 03:02:02: >> run

Here is the template it needs to match:

[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run
[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run
[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run
[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run
[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run
[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run
[% \d{2} %]/[% \d{2} %]/[% \d{2} %] [% \d{2} %]:[% \d{2} %]:[% \d{2} %]: >> run

Any regular expression can be embedded inside the tags, anything which is not inside the tags is a simple text-string. For each line a regular expression is assembled from the parts (with the text-strings being first passed through the quotemeta function. Each line in the input file is then compared with the regular expression generated from the same line in the template.

It can be used something like:

use Test::More tests => 1;
use Test::FuzzyMatch;

is_fuzzy_equal_files( 't/boot.tmpl', 't/boot.log' ,
                                  'checking that the log file is correctly formatted');

This could be achieved by just writing every line in the expected output as a regular expression. I think this is clearer in terms of both reading and writing. It also means that we could optimise for lines where no fuzzy-matching is required.

I’d like to take this idea a bit further and add support for simple loops to handle repetition. The boot logfile example above shows how it would be nice to say something like “the next 7 lines must match this”.


Learning Perl

October 7, 2008

I was going to post this as a comment to Chris’s recent post but it started to get a bit long and involved so I reckoned it would be better as a separate entry on my blog.

The Programming Perl book is basically considered to be the only complete specification of what Perl is and how it works so there is a lot of value in it going into detail for every clever and odd edge case. As Chris said though, it’s definitely not a useful book for learning how to programme in Perl.

In my opinion, the best route is Learning Perl, Intermediate Perl and Mastering Perl, the three books are designed to be thorough introduction to Perl programming. The Perl Cookbook is excellent and invaluable for solving those small problems which others have previously encountered. It is sadly getting rather old now, the most recent edition was published in 2003 which is rather a long time in terms of Perl and currently accepted best practices. The main issue I have with it is that I find it a bit lacking in general advice on how to proceed with some problem which doesn’t fit directly into any of their categories. A more minor issue is that Perl people love the “There’s More Than One Way To Do It” attitude so the way one person solves something may not appeal to someone else.

I’ve recently found the Perl Best Practices (and the related perlcritic tool) to be very useful and I recommend it to anyone who regularly writes a lot of complex Perl code. I believe it has helped improve the quality, readability and maintainability (some would say those 3 things are all the same really) of my code enormously over the last year or so.


Module::Build with XS

September 30, 2008

This is mainly for my own future benefit. It has just taken me ages to discover how to build Perl modules using Module::Build when parts of the module are done with XS.

The Build.PL needs to contain:

    extra_compiler_flags => ['-Wall'],
    extra_linker_flags => ['-lfoo'],
    include_dirs  => ['.'],

where foo is whatever library you need to link against. The include_dirs specifies where to find the ppport.h file. Also, the build-requires must be like:

    build_requires => {
        'ExtUtils::CBuilder' => 0,
   },

Finally, unlike the old MakeMaker, XS modules should go in the right place in the lib/ tree, e.g. Foo::Bar should be lib/Foo/Bar.xs, there will probably also be a lib/Foo/Bar.pm already in that directory.


Platform Independence

September 11, 2008

Platform-independence is something which everyone aspires to when writing code (I hope) but is often one of the first things to go out of the window when time pressures come into play. In the process of preparing the LCFG build tools Perl modules for upload to CPAN I’ve been modifying various chunks to be more platform-agnostic. In particular I’ve been focussing on the file and directory name handling, it’s not that we need the code to work on Windows right now but it’s much more likely to happen at some point in the future if we start out as we mean to go on. When the code was first written I thought it was easiest to just use these sort of idioms:

my $fullpath = join '/', $dirname, $filename;

my @parts = split /\//, $fullpath;
my $file = pop @parts;

It turns out that by ignoring platform-independence I didn’t get all the advantages of excellent modules like File::Spec and File::Basename. For example:

my $fullpath = File::Spec->catfile( $dirname, $filename );

my ($volume,$directories,$file) = File::Spec->splitpath( $path );

I think that code like that is clearer and it is platform-independent without any further effort.

One thing I’ve needed several times is the ability to take two paths and find one as relative to the other. First thoughts might suggest this is simple but I ended up with several different solutions involving regular expressions which coped with them being absolute or relative to begin with. Whilst poking around the perldoc for File::Spec today I noticed:

       abs2rel
         Takes a destination path and an optional base path returns a relative
         path from the base path to the destination path:

             $rel_path = File::Spec->abs2rel( $path ) ;
             $rel_path = File::Spec->abs2rel( $path, $base ) ;

         If $base is not present or ’’, then Cwd::cwd() is used. If $base is
         relative, then it is converted to absolute form using "rel2abs()".
         This means that it is taken to be relative to Cwd::cwd().

         On systems with the concept of volume, if $path and $base appear to
         be on two different volumes, we will not attempt to resolve the two
         paths, and we will instead simply return $path.  Note that previous
         versions of this module ignored the volume of $base, which resulted
         in garbage results part of the time.

That’s exactly what I need and it handles all the cases I need. The code can now be simplified and clarified and only by thinking about platform-independence did I actually discover that function.


LCFG on CPAN

September 11, 2008

The plan for the new LCFG build tools has always been that they should be packaged as “standard” Perl modules. The idea being that this should help people who want to port LCFG to new platforms or just work on platforms other than those directly supported. A direct consequence of this is that they can be uploaded to the Comprehensive Perl Archive Network (CPAN) which vastly improves distribution of the software. Putting Perl modules onto CPAN means they can be easily converted into RPMs using cpanspec, Debian packages using dh-make-perl or just installed directly using the CPAN shell which is shipped as part of the standard perl distribution. I’ve now uploaded most of the build tools packages, only LCFG::Build::Tools is still to go, here’s the links: