The fact of the matter, though, is that applying numbers or at least equations to the process of software development is not that hard. There are a number of concepts that are already out there and are fairly commonly accepted that can be applied toward that goal. Today I want to introduce some of them. I want to take some of these equations and break them down so that we can understand how they operate, and how we operate through them.

Let's begin with Value. Value is a concept we are all familiar with. The things we build or buy have value; our car or bicycle, our meal or beer. We also value things and people based on our judgements about them. Ask someone to explain why something has value to them, though, and you are likely to get a vague answer somewhere between 'I just like it' and a set of personal judgements about the quality of said item.

There will be a thread, though. Value is practically always defined as a relationship between Quality and Cost. This relationship can be (over)simplified to the following equation:

In order for that to have any meaning, though, we need to be able to quantify it. This is not exactly easy to do. There are no units of measurement for Value or Quality. They exist only as perception, and even then are only meaningful when related to something else.

Quality

There is no unit of measurement for Quality. Quality is a vague thing. It is often hard to describe and can be difficult to nail down. I have, however, come up with a definition of quality which I find works in nearly every application of the term. My definition of Quality is "suitability for its purpose." A thing is generally said to be of excellent quality when it performs its purpose flawlessly. When a thing is said to be of low quality, it generally performs its purpose sub-optimally, or not at all. This definition clearly defines Quality as a range and for the range to be useful, an upper and lower numeric bound must be defined.

To keep our concept of Quality grounded, we will use a range of 0 to 1. When judging software using this definition and range of Quality, we will say that a piece of software that performs every aspect of its purpose exactly the way it is intended has a Quality of one (1). A Quality of zero (0), by contrast, does not perform its intended task at all. Obviously, when developing software, the ideal target is always a Quality of 1. Keep this in mind, we will use it later.

Value

Now let's explore the next element of the equation, value. Value is another hard to define concept. There is no unit of measurement that is generally defined for value. It is again a range and is somewhat arbitrary. In order for our results to be meaningful we need to define a range. When we speak of the value of a thing, we may say it has no value, or that it has high value. So, we will use a range that is in keeping with the way we usually use the word. We will say that if a thing has no value, its value is 0. Let us say, for the purposes of this subject that a value of 1 is a 'neutral value' that is neither exceedingly good, nor exceedingly bad. Anything that has a value that is higher than one (1) is of 'high value.'

Cost

Now that we have workable definitions and ranges for Quality and Value, we only have one component of our initial equation left to define: cost. Generally speaking, cost can be quantified in terms of money, or time, or a relation of the two. When developing software, cost is most often quantified using money. The time element of cost can usually be converted to money by multiplying the total hours required by the hourly rate of the developer or developers. Every working developer knows this equation all too well.

So now we have a working definition of cost as well. We have to apply it to our original equation. In order for it to work, however, we have to remember that value has a perceptual element as well. Remember that value is judged based on our concept of what something's cost should be according to some ideal.

Ideal cost

So what is ideal cost? You already know the answer to this. If the actual cost is what it really takes to get a set of features done, then the ideal cost would be the minimum required to get those features working. Let's use a web application as an example. We know that each application has business logic that is unique and specific to that application, it also has presentation code that will be specific to that application. In an ideal world, we would only have to write the code to define that behavior. So our ideal cost is what it would take to write just the business logic and presentation portion of our application.

All of this translates to an equation that looks like this:

With this equation, you have a working model for determining value. With our target Quality as 1, as actual cost approaches ideal cost, our value approaches 1, or 'good value.' If actual cost is less than ideal cost, value increases, and likewise if actual cost is more than ideal cost, your value goes down.

The problem with this equation right now is that it only works if we always use our target of quality level of 1, or highest quality. As any working developer knows, this highest level of quality is rarely attained. Unfortunately, this returns us to the question, how do you define quality with regard to software? We've given it a range, but it currently has no meaning. In order for our overall equation to mean anything, quality must be more clearly defined.

Software Quality Revisited

There are many ways you can judge a code's quality, but many of them are subjective. Here we want to focus on as objective measures as possible. Different people have different preferences, but ultimately there are some objective areas that can be used nearly universally. For our purposes there are two areas that define the quality of a chunk of code.

Functionality

First and most simply is functionality. Functionality simply answers the question 'does this code actually provide the feature required in its entirety?' Functionality is easily determined if you have a complete specification. You can think of functionality as a percentage. Take the implemented number of components of a given feature and divide it by the total components of that same feature according to the spec.

Any professional developer with even a few projects under his belt would agree that overall code quality tends to suffer when there is an incomplete (or worse, nonexistent) spec. It is mathematically impossible to solve the value equation if required functionality remains an unknown.

Note that often people define many other areas to consider when evaluating software quality, for example: reliability, scalability, portability, usability, etc. When it comes down to it, though, each application will have different requirements for each of these areas. In this respect, it is clearer to think of these requirements as feature requirements and thus they fall under the functionality heading in terms of analysis.

Maintainability

The second part of software quality is maintainability. Software rarely is written and never modified, so for any real code quality judgement you have to look at how easy it will be for the code to be maintained and modified down the road. In my experience code's maintainability can be defined as a combination of three elements: documentation, clarity of code and verifiability of functionality.

Documentation we are all familiar with. It is the easiest element of code quality to control, as the author of the code can easily write what the code does in plain language. Sadly, it is all too often neglected. The level of documentation required can be somewhat subjective depending on the reader, but for our purposes we are talking about documentation required for the software to be well maintained. This includes documentation of the intended function, as well as the details of how the code functions. Note that this includes both formal documentation and code comments. We can quantify documentation quality by comparing the information required to maintain the code with the information actually present in the form of documentation. This will vary from developer to developer, but in practice it comes down to something akin to code coverage testing. In my opinion, if the general purpose of the code is defined, and every function or method is documented in terms of what it expects as arguments, what state (if any) it affects while running, and what it produces as a return value, the documentation requirement has been met. This, like functionality, can be taken as a percentage.

Clarity of code is an assessment of the code itself. Code clarity answers the question 'How easy is it to determine what the code is doing?' Are variables well named? Are clever but hard to understand solutions eschewed for clearer, if more verbose, solutions? When hard to understand solutions are unavoidable, are there enough code comments to decipher the associated code? All of this boils down to the question, 'When a change is required to the code, can you easily determine where and how to make that change?' This is a difficult one to quantify, but it is not unreasonable to treat it as a 0/1, yes or no proposition. If another reasonably competent developer (who is unfamiliar with the code) can read it and understand it without requiring additional explanation or a long code spelunking process, then clarity of code is achieved, otherwise it is not.

Verifiability of functionality is the ability to objectively verify that a piece of code does what it is supposed to do. Generally speaking, there is one way to verify functionality: unit tests. In order to achieve maintainable software, you have to be able to tell whether the maintenance change you just did damaged the code. In order to do that, you need to have set inputs and expected outputs and you need to have automated tests that verify that the code produces the expected outputs (and side effects) when a given set of inputs are provided.

If you can not easily verify this for each piece of code your maintenance will be significantly more troublesome. The clearest (and quite significantly discussed) method of evaluating this is 'code coverage' analysis of your tests. Taking your code coverage as a percentage is sufficient for this element.

Complete definition of Quality

So, the quality of a given set of code or feature can be said to be the average of these four elements: functionality, documentation, code clarity and tests. If you think of each of these elements as a percentage, or a fraction of 1, then our quality equation becomes:

Which makes our Value equation:

Boiling it down

We now have equations that give us a way to gauge both the quality of software and its value, but what do we do with them? Well, to begin, anyone in charge of the completion of a software project now has a way to evaluate the completeness of a piece of software. We also have a way to compare competing software packages more objectively. There are likely always to be factors to consider outside of those we have covered. That said, however, at minimum, the quality equation gives us a baseline to evaluate all packages. If the values are significantly skewed at the quality level to start, evaluation of additional factors may be irrelevant.

As developers it is easy to at first be put off by these figures and equations. It can seem like a recipe for more slapping around by managers who don't understand what development really entails. I think, however, that this can have the opposite effect. It provides an unbiased and clear set of criteria for evaluating code quality. It allows you to base your code creation process on factors that do not change, and it provides a clear business reason for defining specifications ahead of time.

These formulas provide a way to explain a target for software development that makes our jobs easier. It also provides a way to explain why the all too common practice of 'We need it now, I don't care how ugly it is' ultimately leads to low quality software and an unmanageable disaster of maintenance.

Obviously, these can only be useful if you can apply them. Next time, I will be covering some of the more interesting implications of these equations. In the meantime, I'd love to hear your experiences with development process, selecting software and modules, and what you've found useful for improving software quality. Got a thought to share? Sound off in the comments.

Web spiders are not a new idea.

A well written spider can gather a whole lot of data without any labor beyond actually writing it, and in a speedy manner.

I've put together a spider for a recent project I've been working on, using Web::Scraper.

So here we go!

We're using Smart::Comments so we don't have to add a bunch of print statements around.

#!/usr/bin/env perl
use Smart::Comments;
use warnings;
use strict;
use Web::Scraper;
use URI;
use YAML qw/ Dump /;
use WWW::Mechanize;
use WWW::Mechanize::Link;
use URI::Query;
use URI::Escape;
use IO::File;

For starters, we grab the URL we want to hit and scrape using WWW::Mechanize.

### On your marks
my $yaml;
my $start_url =  WWW::Mechanize::Link->new( { url =>'your_url_to_scrape'});

### Get Set
my $mech = WWW::Mechanize->new;

Here, we loop through our page, and the subsequent pages if there is pagination.

### GO!
foreach my $l (@letters) {
    my $base_url = $start_url->url;
    $base_url =~ s/__HERE__/currentLetter=$l/;
    my $page = 1;
    ### Letter: $l

    while ($base_url) {
        ### Page: $page
        $mech->get($base_url);
        my $next = $mech->find_link( text_regex => qr/^Next$/i);

        # Bailout
        $base_url = $next ? $next : undef;

        $page++;

        my @gold        = scrape_some('gold', $mech);

        # bailout condition
        undef $base_url  if (!@gold && !@free && !@nearly_free); # nothing on this or subsequent pages for this loop.
        my @information = (@gold, @free, @nearly_free);
        open my $OUT, ">>", "full.yml";
        print $OUT Dump(@information);
        close $OUT;
    }
}

This is the actual Web::Scraper code. Wonderfully, all we have to do is specify some DOM attributes and Web::Scraper will grab the data as we specify. TEXT is the text data in the element being searched for and scraped. @href is the actual link URL.

 
### All done!

sub scrape_some {
    my ( $list_type, $mech ) = @_;
    my @contractors; # return value
    my $want = scraper {
        process "li.$list_type" , "contractors[]" => scraper { 
            process ".listing_link",   name    => 'TEXT';
            process ".address", address => 'TEXT'; # need to split this up into address, state, postcode,
            process ".phone",               phone   => 'TEXT';
            process ".links",                     website => '@href';
        };
    };
    my $ua = $want->user_agent;
    my $names = $want->scrape( $mech->content, $mech->uri);
    my @ppl = ();
    @ppl = @{$names->{data}} if $names->{data};

    foreach my $p (@ppl) {
        if (exists $p->{website}) {
            my $site = $p->{website};
            my $true_url      = URI->new($site);
            my $query = URI::Query->new($true_url->query);
            my $site_from_query = uri_unescape($query->hash_arrayref->{webSite}->[0]);
            $p->{website} = $site_from_query;
        }
        $p->{type} = $list_type;
        push @contractors, @ppl;
    }
    return @contractors;
}

This could definitely be refined, however having not done very many web crawlers in the past, Web::Scraper saved the day for me.

A chart is only useful when it displays some useful data. Most of the time, this data will be coming from a database somehow, perhaps with some adjustment on the way out. In my case, the data source is a set of student test scores stored in a database. I use a DBIx::Class custom ResultSet classes to get at the data, and then munge it a bit to suit my needs. As an example, here is how I get at the averages for each of the five sections of the test for an arbitrary school and grade¹.

sub school_section_averages {
    my $self      = shift;
    my $admin_id  = shift;
    my $school_id = shift;

    my $school_section_scores = $self->search_rs(
        {
            'me.admin_id'             => $admin_id,
            'student2class.school_id' => $school_id,
        },
        {
            'select' => [
                { avg => 'me.percentage_score' }, 'me.section_id',
                'section.name', 'section.position',
            ],
            'as' => [
                'average_score', 'section_id',
                'section_name',  'section_position',
            ],
            'join' => [ 'student2class', 'section' ],
            'group_by' =>
              [ 'me.section_id', 
               'section.name', 
               'section.position', ],
            'order_by' => ['section.position'],
        }
    );

    my @school_section_averages;
    while ( my $section_average = $school_section_scores->next ) {
        push @school_section_averages,
          sprintf( "%.0f", 
                    $section_average->get_column('average_score') );
    }
    return \@school_section_averages;
}

I do something similar to get at section averages by classroom teacher which we'll chart later.

Mediating the Data - The Controller

In a standard Catalyst controller I acquire the underlying data and ship it to a Private action for further processing². This processing is where I create a Perl data structure that defines all the attributes of the chart. This data structure is then transformed into JSON which is the data format required by Open Flash Chart 2³. Let's see what the structure looks like:

    my $chart_data = {
        'bg_colour' => '#ffffff',
        'y_axis'    => {
            'tick_length' => 3,
            'colour'      => '#d000d0',
            'stroke'      => 1,
            'max'         => 100,
            'grid_colour' => '#00ff00',
            'offset'      => 0,
            'steps'       => 10,
        },
        'x_axis' => {
            'colour' => '#d000d0',
            'stroke' => 1,
            'labels' => {
                'labels' => $labels,
                'size'   => 12,
            },
            'grid_colour' => '#00ff00',
            'tick_height' => 10
        },
        'elements' => $elements,
        'title'    => {
            'text' => $title,
            'style' => '{
                             font-size: 20px; 
                             color:black; 
                             font-family: Verdana sans; 
                             text-align: center;
                         }'
        },
        'y_legend' => {
            'text'  => 'Percent Correct',
            'style' => '{color: #736AFF; font-size: 14px;}'
        },
        'tooltip' => {
            'mouse'      => 2,
            'stroke'     => 1,
            'colour'     => '#000000',
            'background' => '#ffffff'
        },
    };

The three variables $labels, $elements and $title have to be constructed as well, but I've left them out for the sake of clarity. Suffice it to say that $elements is the most interesting as it defines the actuall chart values and chart styles. The $elements variable is actually an ArrayRef[HashRef] where the HashRef values could be a scalar or another ArrayRef. To help ease your head from spinning here is example of one element⁴:

{"alpha":"0.5","colour":"#9933CC","text":"Fergie Liscious",
"values":[93,84,92,89,88],"type":"bar","font-size":12}

Once I've packaged up the data into a Perl data structure, I can turn it into JSON with the help of the JSON module like so:

use JSON;
my $json   = new JSON;
my $json_chart_data = $json->encode($chart_data);

Turning Data into a Chart - The View

At this point we have put the data into the format needed by OFC2. Now let's focus on building the view. The approach I take is to construct a web page via Template Toolkit (TT) that has the necessary Flash and Javascript library links along with the JSON data embedded in the page. Here's what the TT template for the page head looks like:

Let's go through this template in parts:

json2.js

This is the Javascript JSON library used to stringify the JSON for OFC2. It uses var data which comes from the JSON data we encoded earlier in the Controller.

swfobject.js

Optional helper Javascript library used to reduce the amount of Javascript code needed in this template.

open-flash-chart-custom.swf

This Flash application is the center-piece of the charting process. It is a custom version of OFC2⁵ It uses the open_flash_chart_data() function to consume the chart data and render the chart. The ofc_averages_chart parameter is the id of an HTML div where the chart will be placed in the web page.

Final View

If you'd like to see an example of the final output, you can go HERE.

As you can see, after this relatively small amount of work, we have a dynamic, flexible and shiny chart tracking the data we are interested in. Open Flash Chart is a nice and simple way to add a bit of 'wow' factor to a set of statistical data, and including it in your Catalyst app is a breeze.

Footnotes

¹ The $admin_id represent a test given to a certain grade level at a certain time. e.g First Grade Spring 20009.

² Yes, I would be wise to put my controller on a diet and move more of the data processing into the model.

³ There are some Perl modules that can assist this process but the mature one is for the older (non-JSON) format, while the newer one is not stable.

⁴ I'm actually showing the element in it's JSON form, but the Perl form is very close, just substitute ':' with ' => '.

⁵ I'm using a slightly modified version of OFC2. It includes a patch that provides keys to shapes. The patch came from DZ. This custom library was compiled with Flex SDK 3.3 on Linux. Bug me for another article if that interests you.

I've returned, with the full source listing of Creating a Simple Server - No HTTPD Required.

So here it is...

• You'll notice, this is a fully functional web application. This essentially replaces and obsoletes a, say, CGI version of this application as well as the web server it attaches to.
• You have a dispatch table you can set up, authentication (via HTTP::Server::Simple sublcasses), SSL (via the commented out SSL hook),
• and a fully embedded server that allows for preforking.

And without further adieu:

lite.pl

#!/usr/bin/env perl

package Server::Lite::App;
use Moose;
use namespace::autoclean;
use Server::Lite;
use IO::File;
use MooseX::Types::Moose qw/Str Int/;

with 'MooseX::Getopt';

has logfile => (
    is => 'ro',
    isa => Str,
    traits => [qw(Getopt)],
    cmd_aliases => 'l',
    documentation => qq{ specify a log name for syslog },
    required => 1,
);

has task_dir => (
    is => 'ro',
    isa => Str,
    traits => [qw(Getopt)],
    cmd_aliases => 'dir',
    documentation => qq{ the directory where task queues are stored },
    required => 1,
);

has pid_file => (
    is => 'ro',
    isa => Str,
    traits => [qw(Getopt)],
    cmd_aliases => 'pid',
    documentation => qq{ name of the pidfile to be written to },
    required => 1,
);

has port => (
    is => 'ro',
    isa => Int,
    traits => [qw(Getopt)],
    cmd_aliases => 'p',
    documentation => qq{ specify a port to listen to },
    required => 1,
);


sub recorder_prefix { # set the log file for recorder
    my $self = shift;
    DateTime->now . $self->port;
}

sub net_server {
    "Net::Server::PreForkSimple";
}

sub bad_request {
    print "HTTP/1.0 404 Bad request\r\n";
}

sub write_pid {
    my ($self, $pid) = @_;
    my $fh = IO::File->new;
    my $pid_file = $self->pid_file;

    if ($fh->open("> $pid_file") ) {
        print $fh "$pid\n";
        undef $fh;
    } else {
        warn("Cannot open: $pid_file: $!");
    }
}

sub graceful_shutdown {
    my ($self, $server) = @_;
    my $pid = $self->pid;

    print "Shutting down...\n";
    $server->logger->log(
        level => "notice",
        message => "TERM received. Shutting down..."
    );
    `rm $pid`;

}

sub init {
    my ($self) = shift;
    ## start the server
    unless (@ARGV) {
        print "usage: perl bin/lite.pl [options]\n";
        exit;
    }

    my $server = Server::Lite->new($self->port);
    my $logger = Log::Dispatch::Syslog->new(
     name => $self->logfile,
     min_level => 'info', );
    $logger->log( level => 'error', message => "data logging initiated");
    $server->logger($logger);
    $server->dir($self->task_dir);
    my $pid = $server->background();
    $self->write_pid($pid);
}

my $server = __PACKAGE__->new_with_options;
$server->init;
$SIG{'TERM'} = sub { __PACKAGE__->graceful_shutdown($server) };

1;

Server.pm

package Server::Lite;

use Moose;
use HTTP::Server::Simple;
use IO::Socket::SSL;
use IO::File;
use Regexp::Common qw /URI/;
use DateTime;
use File::Spec;
use Log::Dispatch::Syslog;
use MooseX::Types::Moose qw/Str Int/;
use namespace::autoclean;

extends qw/HTTP::Server::Simple::CGI/ ;

has logger => (
    is => 'rw',
    isa => 'Log::Dispatch::Output',
    required => 1,
);

has dir => (
    is => 'rw',
    isa => Str,
    required => 1,
);

sub get_dispatch {
     my ($self, $path) = @_;
     my %dispatch = (
        '/do' => \&handle_it,
     );

     return $dispatch{$path};

}

sub handle_request {
    my ($self, $cgi) = @_;

    my $path = $cgi->path_info();
    my $handler = $self->get_dispatch($path);

    if (ref($handler) eq "CODE") {
         print "HTTP/1.0 200 OK\r\n";
         $handler->($self, $cgi);

     } else {
         print "HTTP/1.0 404 Not found\r\n";
         print $cgi->header,
               $cgi->start_html('Not found'),
               $cgi->h1('Not found'),
               $cgi->end_html;

     }

}

#sub accept_hook {
# my $self = shift;
# my $fh = $self->stdio_handle;

# $self->SUPER::accept_hook(@_);

# my $newfh =
# IO::Socket::SSL->start_SSL( $fh,
# SSL_server => 1,
# SSL_use_cert => 1,
# SSL_cert_file => 'myserver.crt',
# SSL_key_file => 'myserver.key',
# )
# or warn "problem setting up SSL socket: " . IO::Socket::SSL::errstr();

# $self->stdio_handle($newfh) if $newfh;
#}

sub handle_it {
    my ($self, $cgi) = @_;

    return if !ref $cgi;

    my $dir = $self->dir;
    my $prefix = $cgi->param('prefix');
    my $goes_in_queue = $cgi->param('to_queue');
    my $to_url = $cgi->param('url');
    my $now = DateTime->now;
    my $activity = File::Spec->catdir($dir, "$goes_in_queue$now");

    print $cgi->header;

    unless ( !$prefix or !$goes_in_queue or $to_url !~ /$RE{URI}{HTTP}/ ) {

        my $fh = new IO::File;
        if ( $fh->open(">$activity") ){

            print $fh "$goes_in_queue" or print $cgi->h1("File IO Error:$!");
            $fh->close;

            $self->logger->log( level => "error", message =>$cgi->remote_addr . "\t" . "URL: $to_url\t" .
                          "Prefix: $prefix \t Command: $goes_in_queue \t Status: Success\n" ); # or die "Error: $!";

        }

        print $cgi->start_html('Success!'),
              $cgi->h1("Successfully handled request"),
              $cgi->p("Dir: " . $self->dir),
              $cgi->end_html;

    } else {

        print $cgi->start_html('Fail!'),
              $cgi->h1("Missing required parameters!"),
              $cgi->end_html;

        $self->logger->log( level => "error", message => $cgi->remote_addr . "\t" . "URL: $to_url\t" .
                      "Prefix: $prefix \t Command: $goes_in_queue \t Status: Failed\n" ) or die "Error: $!";

    }

}

# ABSTRACT: A really simple server + web application implementation with SSL, HTTP authentication *and* preforking options

=head1 NAME

Server::Simple

=cut

=head1 DESCRIPTION

A really simple server + web application implementation with SSL, HTTP authentication *and* preforking options

=cut 

=head1 SYNOPSIS

    perl  bin/lite.pl --pid /tmp/server-lite.pid --logfile local1 --dir tasks/ --port 3001

=cut

=head1 OPTIONS

pid: specify a pidfile for the server

logfile: a valid syslog service to connect to

dir: the dir to which command queue files will be written to

port: a port to listen to 

=cut

=head1 SEE ALSO

Moose, HTTP::Server::Simple, Log::Dispatch::Syslog

=cut

=head1 AUTHOR

Devin Austin , Jay Kuri 

=cut

1;

You can download the source here: tar.gz, zip

This time it was oriented towards Rails and Catalyst, but I've seen it on other projects as well. The question behind the above statement is somewhat deceiving. It seems to be: 'Do I choose a system that has lots of dependencies, or one that has only a few (or none)?' Well, when you put it like that, it seems like a no brainer... but it is, in reality, a trick question.

So what's the difference between something with 'internal dependencies' and something with 'external dependencies'? The difference is that the 'internal dependency' projects try to do all those small tasks themselves, where the 'external dependency' projects delegate those small tasks to external modules specifically built for those particular tasks.

The practical differences between these two are subtle, but important. It's a difference in viewpoint for one. Those in the 'external dependency' camp tend to prefer to focus on the problem they are trying to solve, and use tools to solve the little tasks that are built for solving those little tasks. The 'internal dependency' camp tends to have a more 'do it myself so I can get exactly what I want' type view. Both of these views have merit in certain situations, but ultimately I think the 'external dependency' camp wins for all practical purposes and I'll explain why.

One of the biggest wins for the 'external dependency' camp is that they are focused on their own particular problem and spend less time Yak Shaving, writing Yet Another Templating Language, etc.

Another important fact is that when a module is published on its own, it has to stand up on its own. This generally means more documentation and clearly defined interface. Interfaces are important because they are the contract between the code being used and the application which is using it. A clearly defined API makes the code that uses a module MORE solid, as it is clear what the expectations are on both sides.

Documentation solidifies APIs. When an API is documented, it becomes public and it adds weight to the code, making it harder to modify. When it's documented and published, it gets even more solid. It's like a sign that says 'don't pull this rug, someone might be standing on it'.

When the same small task is accomplished by some chunk of code in some sub-module within the main application, it is often without a clearly defined API and documentation. This doesn't seem like a big deal until you realize that it usually means that the application author feels like he can adjust the calling semantics or return values to work better in the newest release of ApplicationFramework vWhizBang.42. When that happens, any code that you wrote that used that functionality suddenly breaks. If you are lucky the change is documented somewhere so you can find out why your application doesn't work anymore. Most of the time you are not lucky.

Which brings us to the next point; the author of the 'small task' module has incentive to fix the problem without disrupting the API. As I mentioned, once documented and published, a module becomes solid. As soon as it's out there, people start depending on it, and not just the people who wrote it. This forces the author to think hard about making a change to the API. This is not motivated by some fuzzy 'goodness' feeling toward other people, it's simply the background knowledge that 'boy, if I change this, I am going to get swamped with emails and complaints'. All of this means that the author has a reason to fix whatever problem it is without disrupting the API... even if it's a little harder to do.

Another aspect of this is that when a module is published, it gets used by more than one application. It almost always starts getting used for things that the original author didn't intend or didn't think of. Any seasoned developer knows, the vast majority of bugs relate to the code being used in ways the author didn't plan for.

What this means is that those bugs are exposed more quickly. When your code is deployed by many different people in many different ways, you have a much larger group of people stress testing and reporting bugs. This means that published modules tend to have their issues identified and corrected more quickly than an unpublished piece of code within another distribution.

A module that is used in many different applications also has the benefit of expanding its feature-set more quickly. Different applications and frameworks grow and develop at different rates and in different areas. When new requirements come up, the self-contained module will likely be updated to better support it. This will happen, often, without a disruption in the API.

The benefit of this, in its simplest terms, is that when you realize that your application requires some additional functionality, often you will find that the module has already gained that functionality. In the 'internal dependency' camp, in the same situation you instead get to have a nice Yak Shaving Holiday, away from working on your core feature set.

In short, having dependencies allows you to focus on your core functionality... the features that are most important to you. All the while those who wrote your dependency modules are focused on their own core functionality, that which is most important to them. It means that each small component has someone focused on making it as good and as solid as it can be.

There is another aspect to all of this that must be discussed. When your application doesn't include all its small tasks within itself, you are actually MORE able to respond to security and other problems. Most of the time, a security or bug-fix type issue can be isolated to a particular module. Because the API is defined and documented, usually that module can be updated independent of the rest of the application. The bug can usually be fixed without disrupting the rest of the code.

There is another subtle effect here. Because each small-task module is versioned independently, they only need to be updated when they have a problem that needs fixing or when your application requires functionality that is new to that module. This means that you can update your Cookie handling module without needing to update your form handling module.

In the 'internal dependency' camp, all the small-task code is part of the larger package. Now, if you need to update, you have a much bigger problem. Instead of just isolating and adjusting the points where the Cookie module is used, you have to look at every change made to the larger package across the board between the version you have deployed and the version that contains your Cookie bugfix. You then have to go through your code and adjust every other interaction with code that has changed whether it's related to your Cookie bugfix or not. This means every update requires much more work.

It also means that with internal dependencies, when your application is left unmodified for a while, you have a much bigger task when it comes time to bring it up to date... and you have to do so all at once, rather than working with one package update at a time.

All of these things demonstrate that ultimately, external dependencies are the way to go. It lets you focus on your core features and gives you a much more solid base to stand on as you build your own functionality.

All of this said, there are still places where internal dependency options are better. Examples of these might be embedded devices or areas where you have little or no control over your deployment environment such as shared virtual hosts, etc.

Fortunately the world of application deployment has progressed beyond shared virtual-hosts. We now have options such as cloud servers and other virtual environments that give us flexible machines where we get to decide what is deployed and how. For most of us, using external modules is a no-brainer.

Waiting for many modules to install can be annoying. Ultimately, when you realize the real benefits of using external dependencies, you realize that the more you stand on the shoulders of others, the better off you will be.