Apr 7, 2008

Deploying Web Sites using TFS Deployer, PowerShell and FTP

I had a situation recently where I needed to deploy a web site into a production environment from TFS where the only way we could transfer files was via FTP.  In case it's not obvious I didn't want to deploy every time the build ran, but rather, only when the build was declared good.

To do this, I used TFS Deployer (developed by Mitch, Chris, Darren and Geoff of Readify) available from CodePlex. TFS Deployer is a small utility that monitors changes in the build quality of a TFS Team Build and initiates a PowerShell script based on what the change in build quality was.  It runs something like this:

deploy1

A TeamBuild is executed and after completion the build has a quality of "Unexamined"

1. Someone runs some tests, checks the build is OK and decides the build quality can be progressed

2. TFS updates the build quality and raises an alert that the build quality has changed

3. On startup TFS Deployer registers itself as a listener for build quality events and picks up the alert.  It examines the old/new qualities and determines if a PowerShell script should be run. If a script needs to be run then PowerShell is started and the script executed.

4. Results from the script are emailed to a specified email address of distribution list.

clip_image002

 

TFS Deployer Setup

I'll assume that TFS Deployer is installed and set up as per the instructions.  To initiate the PowerShell script we set up a deployment mapping as follows:

    <Mapping xmlns=""
Computer="MyBuildServer"
OriginalQuality="Unexamined"
NewQuality="Released to Production"
Script="DeployToProd.ps1"
NotificationAddress="you.are@here.com" />

That's about it.  Pretty tough :-) Now for something a little harder.


Getting Ready for Deployment via FTP


OK, this is where it could get painful.  PowerShell doesn't have any native FTP support. So the first option you have would be to use the native .NET FTP classes from but that's a real pain because you effectively have to implement your own FTP client in PowerShell.  No thanks!


But don't despair - the open source Indy Project is a project that provides a wrapper and helper functions around for all the FTP calls you might want to make (plus a whole bunch of other low level networking goodness) and they have a .NET version of their library.  Grab a copy of that (and the Mono.Security DLL that goes with it) and you'll have all you really need to get going.


I should also mention that because I'm deploying to a production box I'm assuming that IIS is already set up - after all it's very unlikely that we'd be dropping the IIS virtual directory on every deployment.  We'll just be dropping our files in over the top of whatever is already there.


The PowerShell Script


Time for the script itself - I'll just take this section at a time.  First up, the script is called from TFS Deployer which means we have access to the TFS build data for the build that just had it's quality changed. So we're going to set two variables for the folders where things live.  $loc is the root of the build drop location, and $sourcefiles is where the published web site live (ie the stuff we need to deploy).

Set-Location $TfsDeployerBuildData.DropLocation;
$loc = get-location;
set-location "Mixed Platforms\Release\_PublishedWebSites\";
$sourceFiles = get-location;

Opening and Closing FTP Connections


Next we're going to create a few PowerShell script functions to support the opening and closing of our FTP connections.


When we open the connection we're first going to use the .NET assembly loader to bring in the Indy.Sockets library then use the methods in that to make the connection and then return the FTP connection object from the function.  The close-method is a bit simpler in that we simply wrap the call to the FTP-disconnect. (yes, it's not really required as a function, but you never know when you might want to add logging, error handling, etc).

function Open-FTPConnection($ftphost, $username, $password) {

[void][Reflection.Assembly]::LoadFrom("C:\path\to\Indy.Sockets.dll")
$ftp = new-object Indy.Sockets.FTP
$ftp.Disconnect()
$ftp.Host = $ftphost
$ftp.Username = $username
$ftp.Password = $password
$ftp.Connect()
$ftp.Passive=$true;
return $ftp
}

function Close-FTPConnection($ftp) {
$ftp.Disconnect();
}

Oh - For those not familiar with PowerShell syntax the :: operator lets us call static methods on a type and the [] wrap a type identifier. 


FTP Miscellaneous Functions


Next we define a few more functions just to wrap up some of the basic FTP calls.  Note: Download-FTPFile isn't used - it's just there for your reference.

function Get-FTPCurrentLocation($ftp) {
return $ftp.RetrieveCurrentDir();
}

function Download-FTPFile($ftp, $sourceFileName, $targetDir) {
$ftp.Get($sourceFileName, ($targetDir + $sourceFileName), $true, $false);
}

function Upload-FTPFile($ftp, $sourceFileName, $targetFileName) {
$ftp.Put($sourceFileName, $targetFileName, $false);
}

Nothing overly complex in that.


Getting the Contents of an FTP Folder


Now we get to something a bit more interesting.  Here we're creating a function that iterates over the contents of the FTP location and optionally deleting files as it goes.  It will return a list of sub-folders in the folder for later use.


The function works as follows:


1. Get the directory listing for the current FTP location.


2. Checks if the current directory is the root folder or not - appends a trailing slash if it isn't.


3. For each item in the folder...


3a. Parses the string to get the file name


3b. Checks if the item is a directory or a file (directories have a "d" in their attributes)


3c. For folders we return the full path to the sub-folder.


3d. For files we check the delete flag and nuke the file if it is set.


You'll notice that the $result.add($name) call is cast to a [void].  If we don't do this then when we return from the function we get 2 sets of files.  One from the $result object and one for each file name that was written to the output stream by the $result.add() method call.

function Get-FTPFolders($ftp, [bool]$removeFiles) {
$ls = new-object System.Collections.Specialized.StringCollection;
$result = new-object System.Collections.Specialized.StringCollection;
$ftp.List($ls, "", $true);
$currdir = Get-FTPCurrentLocation($ftp);
if($currdir -ne "/") {
$currdir = $currdir + "/";
}
foreach ($item in $ls)
{
[string[]]$fields=[Regex]::Split($item, " +");
$startField=8; #the file/directory name starts after 8 fields
[string]$name=$currdir;

#make sure we join up file names that were split (ie ones with spaces)
for ($field=$startField; $field -lt $fields.Length; $field++)
{
if ($field -eq $startField)
{
$temp = ""
} else
{
$temp = " "
}
$name += $temp + $fields[$field];
}

if ($item.StartsWith("d"))
{ #directory
[void]$result.Add($name);
}
else
{
if ($item.StartsWith("-")) { #files have '-' as first character
if ($removeFiles)
{
$ftp.Delete($name);
}
}
}
}
return $result
}

Iterating/Deleting the FTP Folder Structure


Next I have two more methods.  One to read through the contents of the FTP folder hierarchy, and one to clear it out.  Both methods are roughly the same with only variations for the call to Get-FTPFolders.

function Get-FTPTree($ftp)
{
$thisdir = Get-FTPCurrentLocation($ftp);
$thisdir;
$subfolders = (Get-FTPFolders $ftp $false);
if ($subfolders -ne $null) {
foreach ($xitem in $subfolders)
{
$ftp.ChangeDir($xitem);
Get-FTPTree($ftp);
}
}
$ftp.ChangeDir($thisdir);
return;
}

function Clean-FTPTree($ftp)
{
$thisdir = Get-FTPCurrentLocation($ftp);
$thisdir;
$subfolders = (Get-FTPFolders $ftp $true);
if ($subfolders -ne $null) {
foreach ($xitem in $subfolders)
{
$ftp.ChangeDir($xitem);
Clean-FTPTree($ftp);
$foldername = $xitem.split("/");
$ftp.ChangeDir($thisdir);
$ftp.RemoveDir($foldername[$foldername.Count - 1]);
}
}
$ftp.ChangeDir($thisdir);
return;
}

Note that in the Clean method we delete folders by stripping the folder name off the end of the full path, stepping back up a level and then calling the removedir method.  It's not pretty, but it works.


Putting it all Together


Now we have everything in place let's actually do what we need to do. Here's what happens:


1. We make the connection to the FTP server.


2. We clean out the existing FTP tree


3. We get a list of all the files we're going to upload.


4. We copy each file individually to the ftp server, creating folders where required (note the filename processing).


5. Just to make sure things are right - we get the contents of the site so we can check the upload worked.


6. Close the connection - we're done!

#Make a connection

$f = Open-FTPConnection "my.ftp.server.com" "ftp_user_account" "ftp_account_password";

write-output "----CLEANING----";
Clean-FTPTree($f);

write-output "----UPLOADING----";

$localfiles = (get-childitem $sourcefiles -r)
foreach ($localfile in $localfiles) {
$remfilename = $localfile.FullName.Replace($sourcefiles.ProviderPath, "");
$remfilename = $remfilename.Replace("\", "/");
if ($localfile.Attributes -eq "Directory") {
Write-Output ("Creating " + $localfile.FullName + ":" + $remfilename);
$f.MakeDir($remfilename);
}
else {
Write-Output (" Uploading " + $localfile.FullName + ":" + $remfilename);
upload-ftpfile $f $localfile.FullName $remfilename
$remfilename;
}
}

write-output "----VERIFYING----";
Get-FTPTree($f);

Close-FTPConnection $f

One thing you might notice is that we use the ProviderPath property of the $sourcefiles variable.  This is because $sourcefiles is a PathInfo object and when TFS Deployer runs the PathInfo will contain a UNC path pointing to a network share - if we just use the Path property then we will get the PowerShell Provider identifier in the string causing the FTP upload to fail.


 


Hopefully this is a good starting point for you if you are trying to do the same thing.

6 comments:

  1. I love it when other people write documentation for me :)

    ReplyDelete
  2. Someone had to :-) You should reference that from your project site!

    ReplyDelete
  3. Hi richard/mitch

    To mitch: it's robert from the folk

    To richard: you don't know me but I'm currently in an engagement with carlson and marketing and I believe you have had an engagement here.

    have you ever had to setup an automated TFS deployment onto a live server that's only accessible from VPN? It would be quite useful I think but also quite hard to have a generic solution as there are different VPN softwares out there but maybe a start is to have a tfs extension that worked for cisco software. *coughs* something CISCO (Insert cisco VPN TFS keywords here for google) should probably look at developing

    ReplyDelete
  4. @robert - I assume you're asking about establishing a VPN connection via powershell? Sorry - nothing from me there. However, if security lets you, is there any reason why you can't just leave one established for the deployment account? The ps script should then work as per usual. It also might avoid the issue of trying to embed vpn login credentials in the powershell script.

    ReplyDelete
  5. Hi Richard,

    is there anyway of restricting TFS users to not be able to see specific build qualities? We want to restrict members of our team from accidently changing a quality and running a deploy.

    Cheers

    ReplyDelete
  6. Hi Richard,

    Would it be possible to set this up to deploy the site/changes to multiple servers at once?

    ReplyDelete