Apr 29, 2008

An Agile Agile Presentation for SBTUG Tomorrow Night

Unfortunately Mitch has come down ill (get better, mate!) and won't be able to present at SBTUG - The Sydney Business & Technology User Group - tomorrow night.  Craig has kindly let me step in and cover for him and I'll be doing my Agile Agile presentation so feel free to come along and be a part of the action.

I hope to see you there, and bring your questions!

P.S. Thanks Craig for the kind words.

Apr 26, 2008

The Whitest Beach In Australia

Another quick holiday update - Wednesday we spent the day in the Cape Le Grande National Park near Esperance in Western Australia, which is apparently home to the whitest beach in Australia at Lucky Bay.  After visiting it I can understand why it's so highly rated; and the other beaches in the park aren't that far behind it.

The water is crystal clear, the ocean is a brilliant blue and the sand is so soft and clean it's almost unbelievable. It actually crunches and squeaks underfoot and clumps up like icing sugar, just fantastic! Plus, if you have a 4WD you can drive out along Luck Bay's 3km of shoreline. Oh, and there's almost no-one around - being 750km from Perth certainly helps. Check out these shots for an idea of what I'm talking about.

Here's my foot seen through thigh deep water - you can't do that at a city beach!

image

And here's a view towards the ocean.  Check out the range of blues - simple amazing!

image

And finally a shot along the shore - as you can see the waves are small enough for me to take the camera into the water without fear of getting wet, and being in a national park there's no development along the coastline.  Just Australia at it's best!

image

Apr 14, 2008

In Scarborough Beach

Holidays are great when you're in a fantastic location.  The family and I are currently in Scarborough Beach, Western Australia have a great, relaxing time.

image 

Check out this wonderful sunset.  You don't see sunsets like this on the east coast (for obvious reasons)

image

Apr 11, 2008

Away for a Few Weeks

Just a quick note to let you all know I'll be on leave for a few weeks in Western Australia driving around the southwest corner of this great country of mine.  I _might_ do an occasional blog post while I'm away but it's more likely that I'll be relaxing and sleeping instead - you can always keep up with what I'm doing via twitter.

How To Fix Problems With Locked Files During a Web Application Build

I had a client recently that was having a big problem with builds on dev machines failing due to files being locked and unable to be deleted. The problem irregular and it was causing the developers to get frustrated and waste a lot of time in stopping and restarting Visual Studio to try and release the locks.

The errors the team were seeing were all similar to this:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Microsoft.Common.targets : warning MSB3061: Unable to delete file "C:\WebApplication\bin\Web.dll". Access to the path 'C:\WebApplication\bin\Web.dll' is denied.


The application itself was a web application, build on XP machines running VS2005 SP1 and set to use IIS as the default web server (i.e. the application virtual directory was pointed at the source code folder).


The team had already checked that the indexing service was off and that antivirus wasn't locking files.  When using the sysinternals utilities to see what other processes were locking those files nothing appeared apart from devenv.exe itself (i.e. visual studio).


The problem as it turned out was in the settings of the virtual directory in IIS6.  The virtual directory still had the "index this location" flag turned on.  It turned out that IIS was placing a temporary lock on these files, even though the web application wasn't being started (ie it was just a compile, not a debug run).  Why this didn't appear in the sysinternals utilities I'm not sure.


Anyway, after turning the "index this location" setting off the file locking problem disappeared and the devs started smiling again.  Happy days :-)

Apr 7, 2008

Deploying Web Sites using TFS Deployer, PowerShell and FTP

I had a situation recently where I needed to deploy a web site into a production environment from TFS where the only way we could transfer files was via FTP.  In case it's not obvious I didn't want to deploy every time the build ran, but rather, only when the build was declared good.

To do this, I used TFS Deployer (developed by Mitch, Chris, Darren and Geoff of Readify) available from CodePlex. TFS Deployer is a small utility that monitors changes in the build quality of a TFS Team Build and initiates a PowerShell script based on what the change in build quality was.  It runs something like this:

deploy1

A TeamBuild is executed and after completion the build has a quality of "Unexamined"

1. Someone runs some tests, checks the build is OK and decides the build quality can be progressed

2. TFS updates the build quality and raises an alert that the build quality has changed

3. On startup TFS Deployer registers itself as a listener for build quality events and picks up the alert.  It examines the old/new qualities and determines if a PowerShell script should be run. If a script needs to be run then PowerShell is started and the script executed.

4. Results from the script are emailed to a specified email address of distribution list.

clip_image002

 

TFS Deployer Setup

I'll assume that TFS Deployer is installed and set up as per the instructions.  To initiate the PowerShell script we set up a deployment mapping as follows:

    <Mapping xmlns=""
Computer="MyBuildServer"
OriginalQuality="Unexamined"
NewQuality="Released to Production"
Script="DeployToProd.ps1"
NotificationAddress="you.are@here.com" />

That's about it.  Pretty tough :-) Now for something a little harder.


Getting Ready for Deployment via FTP


OK, this is where it could get painful.  PowerShell doesn't have any native FTP support. So the first option you have would be to use the native .NET FTP classes from but that's a real pain because you effectively have to implement your own FTP client in PowerShell.  No thanks!


But don't despair - the open source Indy Project is a project that provides a wrapper and helper functions around for all the FTP calls you might want to make (plus a whole bunch of other low level networking goodness) and they have a .NET version of their library.  Grab a copy of that (and the Mono.Security DLL that goes with it) and you'll have all you really need to get going.


I should also mention that because I'm deploying to a production box I'm assuming that IIS is already set up - after all it's very unlikely that we'd be dropping the IIS virtual directory on every deployment.  We'll just be dropping our files in over the top of whatever is already there.


The PowerShell Script


Time for the script itself - I'll just take this section at a time.  First up, the script is called from TFS Deployer which means we have access to the TFS build data for the build that just had it's quality changed. So we're going to set two variables for the folders where things live.  $loc is the root of the build drop location, and $sourcefiles is where the published web site live (ie the stuff we need to deploy).

Set-Location $TfsDeployerBuildData.DropLocation;
$loc = get-location;
set-location "Mixed Platforms\Release\_PublishedWebSites\";
$sourceFiles = get-location;

Opening and Closing FTP Connections


Next we're going to create a few PowerShell script functions to support the opening and closing of our FTP connections.


When we open the connection we're first going to use the .NET assembly loader to bring in the Indy.Sockets library then use the methods in that to make the connection and then return the FTP connection object from the function.  The close-method is a bit simpler in that we simply wrap the call to the FTP-disconnect. (yes, it's not really required as a function, but you never know when you might want to add logging, error handling, etc).

function Open-FTPConnection($ftphost, $username, $password) {

[void][Reflection.Assembly]::LoadFrom("C:\path\to\Indy.Sockets.dll")
$ftp = new-object Indy.Sockets.FTP
$ftp.Disconnect()
$ftp.Host = $ftphost
$ftp.Username = $username
$ftp.Password = $password
$ftp.Connect()
$ftp.Passive=$true;
return $ftp
}

function Close-FTPConnection($ftp) {
$ftp.Disconnect();
}

Oh - For those not familiar with PowerShell syntax the :: operator lets us call static methods on a type and the [] wrap a type identifier. 


FTP Miscellaneous Functions


Next we define a few more functions just to wrap up some of the basic FTP calls.  Note: Download-FTPFile isn't used - it's just there for your reference.

function Get-FTPCurrentLocation($ftp) {
return $ftp.RetrieveCurrentDir();
}

function Download-FTPFile($ftp, $sourceFileName, $targetDir) {
$ftp.Get($sourceFileName, ($targetDir + $sourceFileName), $true, $false);
}

function Upload-FTPFile($ftp, $sourceFileName, $targetFileName) {
$ftp.Put($sourceFileName, $targetFileName, $false);
}

Nothing overly complex in that.


Getting the Contents of an FTP Folder


Now we get to something a bit more interesting.  Here we're creating a function that iterates over the contents of the FTP location and optionally deleting files as it goes.  It will return a list of sub-folders in the folder for later use.


The function works as follows:


1. Get the directory listing for the current FTP location.


2. Checks if the current directory is the root folder or not - appends a trailing slash if it isn't.


3. For each item in the folder...


3a. Parses the string to get the file name


3b. Checks if the item is a directory or a file (directories have a "d" in their attributes)


3c. For folders we return the full path to the sub-folder.


3d. For files we check the delete flag and nuke the file if it is set.


You'll notice that the $result.add($name) call is cast to a [void].  If we don't do this then when we return from the function we get 2 sets of files.  One from the $result object and one for each file name that was written to the output stream by the $result.add() method call.

function Get-FTPFolders($ftp, [bool]$removeFiles) {
$ls = new-object System.Collections.Specialized.StringCollection;
$result = new-object System.Collections.Specialized.StringCollection;
$ftp.List($ls, "", $true);
$currdir = Get-FTPCurrentLocation($ftp);
if($currdir -ne "/") {
$currdir = $currdir + "/";
}
foreach ($item in $ls)
{
[string[]]$fields=[Regex]::Split($item, " +");
$startField=8; #the file/directory name starts after 8 fields
[string]$name=$currdir;

#make sure we join up file names that were split (ie ones with spaces)
for ($field=$startField; $field -lt $fields.Length; $field++)
{
if ($field -eq $startField)
{
$temp = ""
} else
{
$temp = " "
}
$name += $temp + $fields[$field];
}

if ($item.StartsWith("d"))
{ #directory
[void]$result.Add($name);
}
else
{
if ($item.StartsWith("-")) { #files have '-' as first character
if ($removeFiles)
{
$ftp.Delete($name);
}
}
}
}
return $result
}

Iterating/Deleting the FTP Folder Structure


Next I have two more methods.  One to read through the contents of the FTP folder hierarchy, and one to clear it out.  Both methods are roughly the same with only variations for the call to Get-FTPFolders.

function Get-FTPTree($ftp)
{
$thisdir = Get-FTPCurrentLocation($ftp);
$thisdir;
$subfolders = (Get-FTPFolders $ftp $false);
if ($subfolders -ne $null) {
foreach ($xitem in $subfolders)
{
$ftp.ChangeDir($xitem);
Get-FTPTree($ftp);
}
}
$ftp.ChangeDir($thisdir);
return;
}

function Clean-FTPTree($ftp)
{
$thisdir = Get-FTPCurrentLocation($ftp);
$thisdir;
$subfolders = (Get-FTPFolders $ftp $true);
if ($subfolders -ne $null) {
foreach ($xitem in $subfolders)
{
$ftp.ChangeDir($xitem);
Clean-FTPTree($ftp);
$foldername = $xitem.split("/");
$ftp.ChangeDir($thisdir);
$ftp.RemoveDir($foldername[$foldername.Count - 1]);
}
}
$ftp.ChangeDir($thisdir);
return;
}

Note that in the Clean method we delete folders by stripping the folder name off the end of the full path, stepping back up a level and then calling the removedir method.  It's not pretty, but it works.


Putting it all Together


Now we have everything in place let's actually do what we need to do. Here's what happens:


1. We make the connection to the FTP server.


2. We clean out the existing FTP tree


3. We get a list of all the files we're going to upload.


4. We copy each file individually to the ftp server, creating folders where required (note the filename processing).


5. Just to make sure things are right - we get the contents of the site so we can check the upload worked.


6. Close the connection - we're done!

#Make a connection

$f = Open-FTPConnection "my.ftp.server.com" "ftp_user_account" "ftp_account_password";

write-output "----CLEANING----";
Clean-FTPTree($f);

write-output "----UPLOADING----";

$localfiles = (get-childitem $sourcefiles -r)
foreach ($localfile in $localfiles) {
$remfilename = $localfile.FullName.Replace($sourcefiles.ProviderPath, "");
$remfilename = $remfilename.Replace("\", "/");
if ($localfile.Attributes -eq "Directory") {
Write-Output ("Creating " + $localfile.FullName + ":" + $remfilename);
$f.MakeDir($remfilename);
}
else {
Write-Output (" Uploading " + $localfile.FullName + ":" + $remfilename);
upload-ftpfile $f $localfile.FullName $remfilename
$remfilename;
}
}

write-output "----VERIFYING----";
Get-FTPTree($f);

Close-FTPConnection $f

One thing you might notice is that we use the ProviderPath property of the $sourcefiles variable.  This is because $sourcefiles is a PathInfo object and when TFS Deployer runs the PathInfo will contain a UNC path pointing to a network share - if we just use the Path property then we will get the PowerShell Provider identifier in the string causing the FTP upload to fail.


 


Hopefully this is a good starting point for you if you are trying to do the same thing.

Apr 1, 2008

Deming's Rules of Management and Agile Methods

I saw a post this morning by David Anderson this morning on how a fault tolerant culture helps create great teams.  Admittedly he was talking about the British cycling team's recent gold medal haul but the point is valid regardless.  In his post he referred to W. Edwards Deming whom I'd never heard of as a source for the principle of removing fear from a culture.  Curious, I flicked over to the wikipedia entry and had a look at what the esteemed but unheard of Deming had to say.

Here's an excerpt from the wikipedia entry:

Deming's 14 points

Deming offered fourteen key principles for management for transforming business effectiveness. The points were first presented in his book Out of the Crisis (p. 23-24)[19].

  1. Create constancy of purpose toward improvement of product and service, with the aim to become competitive and stay in business, and to provide jobs.
  2. Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.
  3. Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.
  4. End the practice of awarding business on the basis of price tag. Instead, minimize total cost. Move towards a single supplier for any one item, on a long-term relationship of loyalty and trust.
  5. Improve constantly and forever the system of production and service, to improve quality and productivity, an thus constantly decrease cost.
  6. Institute training on the job.
  7. Institute leadership (see Point 12 and Ch. 8 of "Out of the Crisis"). The aim of supervision should be to help people and machines and gadgets to do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.
  8. Drive out fear, so that everyone may work effectively for the company. (See Ch. 3 of "Out of the Crisis")
  9. Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and in use that may be encountered with the product or service.
  10. Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.
  11. a. Eliminate work standards (quotas) on the factory floor. Substitute leadership.
    b. Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership.
  12. a. Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.
    b. Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objective (See CH. 3 of "Out of the Crisis").
  13. Institute a vigorous program of education and self-improvement.
  14. Put everybody in the company to work to accomplish the transformation. The transformation is everybody's work.

What struck me as I read this was how much lean and agile processes try to utilise the principles.

Here's my quick take on alignment between Deming's principles and agile methods:

Point 1 - Constancy of purpose.  Best described by the agile manifesto.  It's a clear and unchanging vision of what agile methods are all about and what they are trying to achieve.

Point 2 - Take on leadership for change.  Agile understands and accepts change.  Change happens - let's make the most of it and lead the way towards change.  Change for the better.

Point 3 - Build quality into the product from the start.  Test Driven Development (see video) is a perfect example of this as is the desire to minimise code debt.

Point 4 - Minimize cost.  Time is the number one cost in software development.  Minimizing wasted effort is the single best way to reduce cost.  Minimal code debt also goes a very long way to achieving this

Point 5 - Improve constantly.  Agile calls this principle "inspect and adapt". It's embodied in scrum retrospectives and other review processes.

Point 6 - Training on the job.  For developers this is encapsulated by pair programming.

Point 7 - Institute leadership.  Leadership in an agile project arises from within a team - it is not imposed through management hierarchies (and imposed "leadership" is typically psuedo-leadership - leaders still naturally arise in teams).

Point 8 - Drive out fear.  I don't know that this is explicitly addressed by any agile methodology per se but it is something that all good leaders will endeavour to build into their teams.

Point 9 - Break down barriers.  Agile methods cry out for openness and collaboration.  It's at the heart of what agile is all about.  Barriers are impediments and slow down teams.

Point 10  Eliminate slogans, targets, etc.  The essence of this is that management can't drive people to work harder to improve because of organisational impediments (the system). In software development "the system" is the organisation and the red-tape and processes that organisation imposes on their staff.  Scrum in particular addresses this (indirectly) through the role of Scrum Master - a person responsible for removing impediments to the team.  If the organisation is slow the Scrum Master should be addressing that.

Point 11 - Substitute Leadership.  See point 7.

Point 12 - Remove barriers.  Covered in points 9 & 10.

Point 13 - Self improvement.  Once again the inspect and adapt principle at play - good teams should be applying this personally as well as at a team level.

Point 14 - Everyone works to achieve transformation.  The is the very essence of an organisation with an agile heart.  Everyone working together to achieve an objective and improve what they do.

 

I think I might have to go and hunt for Deming's book and have a read.  I'd be interested in your thoughts as well - especially if you've read Deming's work.