Invoke-Maven

I’ve slowly been building up a personal module at work that I use in my Jenkins workflows.  One function that I wrote a while ago but didn’t get around to using until yesterday was Invoke-Maven.

<#
    .SYNOPSIS
        Executes Maven.
    
    .DESCRIPTION
        Assumes Maven is correctly setup in the PATH, as well as JAVA_HOME and M2_HOME variables.
    
    .PARAMETER goal
        Specifies the goals to execute, such as "clean", "install", or "deploy".
    
    .PARAMETER pomPath
        Path to the POM file. Path is validated with Test-Path.
    
    .PARAMETER deployPath
        Path for Maven to deploy to. Path is validated with Test-Path.
    
    .PARAMETER logPath
        Path to write maven log file to. If this parameter is used there is no console output.
    
    .PARAMETER X
        Turn Maven debugging on.

    .EXAMPLE
        Invoke-Maven -goal deploy -pomPath "C:\Development\Java\Projects\pom.xml" -deployPath "C:\Development\Builds"
#>
function Invoke-Maven {
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory = $true)]
        [ValidateSet('clean', 'install', 'deploy', IgnoreCase = $true)]
        [string]$goal,
        [Parameter(Mandatory = $true)]
        [ValidateScript({ Test-Path $_ })]
        [string]$pomPath,
        [ValidateScript({ Test-Path $_ })]
        [Alias('url', 'deployUrl')]
        [string]$deployPath,
        [string]$logPath,
        [switch]$X
    )
    
    If ($PSBoundParameters.ContainsKey('logPath') -eq $true) {
        If ($goal -match "deploy") {
            $cmd = "mvn -B -f $pomPath $goal ""-Ddeployment.DeployURL=$deployPath"" - l $logPath"
        } else {
            $cmd = "mvn -B -f $pomPath $goal -l $logPath"
        }
        If ($X) {
            $cmd = "$cmd -X"
        }
    } else {
        If ($goal -match "deploy") {
            $cmd = "mvn -B -f $pomPath $goal ""-Ddeployment.DeployURL=$deployPath"""
        } else {
            $cmd = "mvn -B -f $pomPath $goal"
        }
        If ($X) {
            $cmd = "$cmd -X"
        }
    }
    
    Write-Verbose "Invoke-Expression $cmd"
    Invoke-Expression "$cmd"
}

The only real problem with it is that if you use the logPath parameter, Maven doesn’t display anything to the host console.  What I ended up doing in my Jenkins job that uses this function is calling it with Start-Transcript to get both a log file and the console output.

Posted in DevOps | Tagged , | Leave a comment

Creating a PowerShell [hashtable] from a Jenkins text parameter

In Jenkins parameters are funky. In fact, the checkbox for “This build is parameterized” is pretty much a lie from a PowerShell standpoint. What is should say is “This build has environment variables created for it” because that is what it does. When you create a string parameter in a Jenkins job and give it a name of “rootPath” with a value “C:\”, what you get is an environment variable that you need to get at via $env:rootPath. Another problem with “This build is parameterized” is the limited data types you can create… which I recently found out when I wrote a script that used a [hashtable] param and I wanted to host it in Jenkins.  The trick? Use a “Text Parameter” and learn to love ConvertFrom-StringData.

As an example, say I want a $projects hashtable with the title of the project as the key and the variant I’m building as the value… I create a Text Parameter in Jenkins like so:

text_param_hash

Then in the PowerShell script I cast the $projects variable as a hashtable and give it a default vaule that puts the projects environment variable into it:

param(
[hashtable]$projects = $(ConvertFrom-StringData -StringData $env:projects)
)

Write-Output $projects

And then you get yourself a nice hashtable 🙂

[PowerShellTesting] $ powershell.exe "& 'C:\Windows\TEMP\hudson6969899475438348796.ps1'"

Name Value
---- -----
SQL DevInt.Q2.2015.SQL
Binaries DevInt.Q2.2015.Binaries
JBoss DevInt.Q2.2015.Java
ColdFusion DevInt.Q2.2015.CF

Finished: SUCCESS

Posted in DevOps | Tagged , , | 1 Comment

Jenkins on Windows (and more to come)

I recently took over ownership of the Jenkins server that manages the DevInt and Alpha environments at work and had a blast working a hectic server migration. On top of everything moving around, the environments went from having one server each for SQL, ColdFusion, and JBoss to everything being built out as a cluster with various services spread across it. Some refactoring was due and I hope to have time to write more about the things I’ve learned about running Jenkins on Windows in the future. Unfortunately, after doing the work, raising three kids (two of which are still in diapers and destroying everything they touch), and keeping myself on a constant cycle of learning new things… well, blogging took a back seat.

Posted in DevOps, Musing | Tagged | Leave a comment

Converting audio files en masse

We recently got a new (used) family van that has a 40 gig hard drive in the dash to store MP3 files. Problem is, a lot of my audio files were stored as M4A, which the van doesn’t have a codex for. On top of that, my wife just got a karaoke machine for the kids that also only plays MP3 format. So, I wrote a script that uses FFMPEG to do the file conversion and then uses TagLib to copy over the ID3 tag info (because if a file isn’t ID3 tagged, I don’t want it).

[CmdletBinding()]
Param (
    [Parameter(Mandatory=$True)]
    [string]$rootPath
)

$scriptRoot = "$(Split-Path -Parent $MyInvocation.MyCommand.Definition)"
[system.reflection.assembly]::loadfile("$scriptRoot\taglib-sharp.dll")

function ConvertToMp3 ([string]$oldFile) {
    # FFMPEG Documentation
    # https://www.ffmpeg.org/ffmpeg.html
    [string] $ffmpeg = '$scriptRoot\ffmpeg.exe'
    $newFile = $oldFile.Replace(".m4a", ".mp3")
    & $ffmpeg -i "$oldFile" "$newFile" -y
    DuplicateId3Tags $oldFile $newFile
}

function DuplicateId3Tags {
    #TagLib API Documentation - http://taglib.github.io/api/index.html
    Param (
        [string]$oldTag,
        [string]$newTag
    )

    $sourceM4a = [TagLib.File]::Create($oldTag)
    $targetMp3 = [TagLib.File]::Create($newTag)
    #http://taglib.github.io/api/classTagLib_1_1Tag.html#pub-static-methods
    [TagLib.Tag]::Duplicate($sourceM4a.Tag, $targetMp3.Tag, $true)
    $targetMp3.Save()
}

function ConvertAllToMp3 ([string] $sourcePath) {
    $files = Get-ChildItem "$sourcePath\*" -recurse -include *.m4a
    ForEach ($file in $files) {
        ConvertToMp3 $file.FullName
        # Uncomment to remove old file
        # Remove-Item -Path $file.FullName -Force 
    }
}

ConvertAllToMp3 $rootPath
Posted in Programming | Tagged , , , , , | Leave a comment

Multi-threading Maven builds with PowerShell Jobs and a .NET Queue

I was recently given a copy of the script used on our Jenkins server to deploy our Java code base.  My boss wanted to get it working so devs could have a script to compile code on their local sandboxes, rather than wait for the next Jenkins run.  Honestly, I think I spent more time learning how Maven (the Java compiler) works than I did taking the script and changing all the paths around for local use.

So, the thing about working for a guy that knows more than you do is you constantly challenge  yourself to figure out how to make everything better; because you’re pretty sure he could do it better himself if he only had the time.  I knew exactly what he was going to say when I told him it took 45 minutes for the script to run through the list of projects and complete.  “Can you multithread that?”  I’d heard of PSJobs and that you could use them to spawn scripts, but hadn’t ever looked into them before.  So I said “I’ll get back to you on that” and starting researching.

I should say, my early attempts to do this were laughable.  I read up on PSJobs, took the code I’d already put together and wrapped it up in a ScriptBlock for Start-Job to spawn and ran it.  It worked so well it crashed my system!  Turns out my laptop didn’t want to run 40 instances of Maven all at the same time.  Oops.

Back to the drawing board and I found this Scripting Guy post: Avoid Overload by Scaling and Queuing PowerShell Background Jobs.  Genius!  Create a .NET Queue, stuff my paths into it and then pull one out on each run.  This was the first time I’d ever seen an event listener in a PowerShell script that wasn’t a GUI, pretty cool.  Everywhere else I’d read to just start a job, then use Wait-Job and Receive-Job.  This gave me asynchronicity, making it easier for me to check the log file for errors and retry the Maven task on the fly.

#Requires -Version 3.0
#v3 is needed to use PSJobs.

#Load the Forms assembly for use with Select-FolderDialog
[void][reflection.assembly]::Load("System.Windows.Forms, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089")

$Global:VerbosePreference = 'Continue'
$Global:MaxJobs = 3 #number of threads
$Global:MaxRetry = 3 #retry Maven on failure

#Function for Maven Installs required before running deploys
Function Global:BuildDefaultArtifactsFromQueue {  
    Write-Verbose ("Queue Count: {0}" -f $queue.count)
    $pomFile = $queue.Dequeue()
    #path to current working project
    $path=$pomFile.replace("\pom.xml","")
    $project = $path -match "\w+$"
    #get project folder name
    $project = $matches[0]
    
    #create job
    $job = Start-Job -Name $project -ScriptBlock { 
        Param($pomFile, $deployURL, $logPath, $path, $project) 
        $VerbosePreference = 'Continue'
        If(Test-Path $pomFile) {
            Set-Location $path
            Write-Verbose "Working on $project"
            #run Maven and pipe output to a log file
            & mvn clean | Out-File -FilePath "$logPath\install_$project.txt" -Append
            & mvn install | Out-File -FilePath "$logPath\install_$project.txt" -Append
            For ( $i = 0; $i -lt $MaxRetry; $i++ ) {
                #check for errors in the log and retry if found
                If (Select-String -Path "$logPath\install_$project.txt" -pattern "BUILD FAILURE") {
                    Remove-Item -Path "$logPath\install_$project.txt" -Force
                    Write-Verbose "ERROR WHILE INSTALLING $project - ATTEMPT $($i+1)!"
                    Set-Location $path
                    & mvn clean | Out-File -FilePath "$logPath\install_$project.txt" -Append
                    & mvn install | Out-File -FilePath "$logPath\install_$project.txt" -Append
                } Else { continue }
            }
            #warn if errors are still found
            If (Select-String -Path "$logPath\install_$project.txt" -pattern "BUILD FAILURE") {
                Write-Verbose "$project failed to install."
                Write-Verbose "Review - $logPath\install_$project.txt"
            }
        } Else { 
            Write-Verbose "BUILD FAILURE - $pomFile is missing"
        }
    } -ArgumentList $pomFile, $deployURL, $logPath, $path, $project

    #watch the job state and queue the next one on exit
    Register-ObjectEvent -InputObject $job -EventName StateChanged -Action { 
        $results = Receive-Job -Job $eventsubscriber.sourceobject 
        Write-Verbose "Removing event for $($eventsubscriber.sourceobject.Name)"            
        Remove-Job -Job $eventsubscriber.sourceobject 
        Unregister-Event $eventsubscriber.SourceIdentifier 
        Remove-Job -Name $eventsubscriber.SourceIdentifier 
        If ($queue.count -gt 0 -OR (Get-Job)) { 
            BuildDefaultArtifactsFromQueue 
        } ElseIf (!(Get-Job)) { 
            #at this point the installs should be done and deploys can be queued
            $End = New-Timespan $Start (Get-Date)                     
            Write-Verbose "$('Completed BuildDefaultArtifactsFromQueue in: {0}' -f $End)"
            $InputObject = $Global:BuildPOM
            ForEach($item in $InputObject) { 
                $item = "$sandboxURL\$item"
                Write-Verbose "Adding $item to queue"
                $queue.Enqueue($item) 
            } 
            For( $i = 0; $i -lt $Global:MaxJobs; $i++ ) { 
                BuildArtifactsFromQueue 
            }
        }
    } | Out-Null 
    Write-Verbose "Created Event for $project"
} 

Function Global:BuildArtifactsFromQueue {  
    Write-Verbose ("Queue Count: {0}" -f $queue.count)    
    $pomFile = $queue.Dequeue() 
    $path=$pomFile.replace("\pom.xml","")
    $project = $path -match "\w+$"
    $project = $matches[0]
    
    $job = Start-Job -Name $project -ScriptBlock { 
        Param($pomFile, $deployURL, $logPath, $path, $project) 
        $VerbosePreference = 'Continue'
        If(Test-Path $pomFile) {
            Set-Location $path
            Write-Verbose "Working on $project"
            & mvn clean | Out-File -FilePath "$logPath\deploy_$project.txt" -Append
            & mvn deploy "-Ddeployment.DeployURL=$deployURL" | Out-File -FilePath "$logPath\deploy_$project.txt" -Append
            For ( $i = 0; $i -lt $MaxRetry; $i++ ) {
                If (Select-String -Path "$logPath\deploy_$project.txt" -pattern "BUILD FAILURE") {
                    Remove-Item -Path "$logPath\deploy_$project.txt" -Force
                    Write-Verbose "ERROR WHILE DEPLOYING $project - ATTEMPT $($i+1)!"
                    Set-Location $path
                    & mvn clean | Out-File -FilePath "$logPath\deploy_$project.txt" -Append
                    & mvn deploy "-Ddeployment.DeployURL=$deployURL" | Out-File -FilePath "$logPath\deploy_$project.txt" -Append
                }
            }
            If (Select-String -Path "$logPath\deploy_$project.txt" -pattern "BUILD FAILURE") {
                Write-Verbose "$project failed to deploy."  
                Write-Verbose "Review -  $logPath\deploy_$project.txt"
            }
        } Else { 
            Write-Verbose "BUILD FAILURE - $pomFile is missing"
        }
    } -ArgumentList $pomFile, $deployURL, $logPath, $path, $project

    Register-ObjectEvent -InputObject $job -EventName StateChanged -Action { 
        $results = Receive-Job -Job $eventsubscriber.sourceobject 
        Write-Verbose "Removing Event for: $($eventsubscriber.sourceobject.Name)"            
        Remove-Job -Job $eventsubscriber.sourceobject 
        Unregister-Event $eventsubscriber.SourceIdentifier 
        Remove-Job -Name $eventsubscriber.SourceIdentifier 
        If ($queue.count -gt 0 -OR (Get-Job)) { 
            BuildArtifactsFromQueue 
        } ElseIf (!(Get-Job)) { 
            #TODO: Copy JAR files to JBoss
            $End = New-Timespan $Start (Get-Date)                     
            Write-Verbose "$('Completed BuildArtifactsFromQueue in: {0}' -f $End)"
            $ErrorLogs = Get-ChildItem -Path $logPath -recurse | Select-String -pattern "BUILD FAILURE" | group path | select name
            [void][System.Windows.Forms.MessageBox]::Show("$('Completed builds in: {0}' -f $End)`n`nSee error logs:`n$($ErrorLogs.Name)","Done")
        }            
    } | Out-Null 
    Write-Verbose "Created Event for $project" 
} 

Function Select-FolderDialog ($initialDirectory) {
    $OpenFolderDialog = New-Object System.Windows.Forms.FolderBrowserDialog -Property @{SelectedPath=$initialDirectory}
    $OpenFolderDialog.ShowDialog() | Out-Null
    return $OpenFolderDialog.SelectedPath
}

$Global:Start = Get-Date
$now = Get-Date -format "dd-MMM-yyyy HHmm" 
$Global:logPath = "C:\Development\Scripts\Maven\logs\$now"
#list of POM files for Maven Installs
$Global:DefaultArtifactBuildPOM = Get-Content "DefaultArtifactBuildPOM.txt"
#list of POM files for Maven Deploys
$Global:BuildPOM = Get-Content "ParentPOMValues.txt"
$Global:sandboxURL = Select-FolderDialog -initialDirectory "C:\Development\Sandbox"
$Global:deployURL = "C:\Development\Deploy"

mkdir $logPath -f | Out-Null 
#delete old deploys if they exist
If (!(Test-Path $Global:deployURL)) {
    mkdir $Global:deployURL -f | Out-Null
} Else { Remove-Item $Global:deployURL\* -force -recurse }
If (Test-Path C:\Users\$env:username\.m2) {
    Get-ChildItem -Path C:\Users\$env:username\.m2 | Remove-Item -force -recurse
}

$Global:queue = [System.Collections.Queue]::Synchronized( (New-Object System.Collections.Queue) ) 

$InputObject = $Global:DefaultArtifactBuildPOM
#queue up the first list
ForEach($item in $InputObject) {
    $item = "$sandboxURL\$item"
    Write-Verbose "Adding $item to queue"
    $queue.Enqueue($item) 
} 

#spawn the first set of jobs
For( $i = 0; $i -lt $Global:MaxJobs; $i++ ) { 
    BuildDefaultArtifactsFromQueue
}
Posted in DevOps, Programming | Tagged , , , , , | 2 Comments

Modifying Chocolatey and Boxstarter packages for internal use

Reference Note:  I wrote this assuming the reader has an understanding of NuGet, Chocolatey, and Boxstarter technologies (because why else would you want to build your own if you didn’t know what is was), but just in case, here are some links to check out:
http://docs.nuget.org/
https://github.com/chocolatey/chocolatey/wiki
http://boxstarter.org/WhyBoxstarter

Earlier I made a post on using Boxstarter to setup a Chocolatey repository. It’s a very useful tool for being able to publish your own set of packages. And with my MSI packaging background, I can make custom installers for licensed software with everything ready to go. But say you need to take the repository a step further and set it up inside an isolated system; to do that you’ll need to build your own installers for Chocolatey and Boxstarter that configures them to only use the internal repo.

Server Config

For starters we’ll need to add two mime types to IIS, one for ps1 files and another for msu packages. There are several ways you can get that done (WMI and appcmd come to mind), but since the web.config file that comes with the chocolatey.server package already has a mime type for nupkg files, I went ahead and added them there. Scroll down to line 32 in the file and you’ll see the nupkg entry. Add two more entries:
<mimeMap fileExtension=”.ps1″ mimeType=”text/plain”/>
<mimeMap fileExtension=”.msu” mimeType=”application/octet-stream”/>

While digging through web.config, there are a few things to make note of under the appSettings section. First is the apiKey. By default it is set to “chocolateyrocks”, but feel free to change it to anything. Or, another option is to set requireApiKey to false and not worry about it at all. The rest of the options, like packagesPath, may be worth playing with as well, but aren’t in the scope of what I’m doing here.

Chocolatey Installer

Grab a copy of https://chocolatey.org/install.ps1 and place it in wwwroot. Crack it open and make two quick changes. First look for $url at line 20 and set it to ‘http://SERVER/chocolatey.0.9.8.28.nupkg&#8217;. Next, locate where 7za.exe is downloaded at line 44 and change the server path there as well. While we’re talking about 7za.exe, grab a copy of it and place it at wwwroot.

Next up, building the custom chocolatey nupkg file. Start by copying C:\ProgramData\chocolatey\lib\chocolatey.0.9.8.28 to a working directory, I’ll use C:\Temp\chocolatey.0.9.8.28 for this write up. Get a copy of the Chocolatey nuspec file and place it at C:\Temp\chocolatey.0.9.8.28. Then open C:\Temp\chocolatey.0.9.8.28\tools\chocolateyInstall\chocolatey.config and change ‘https://chocolatey.org/api/v2/&#8217; to ‘http://SERVER/chocolatey/&#8217;.

Important note, the last / in the value must be there or package installs from the server will not work correctly. Also, if you don’t use the chocolatey.server package and instead compile your own NuGet server in Visual Studio (like I did on my first test build), the path will be ‘http://SERVER/nuget/&#8217;, instead of ‘http://SERVER/chocolatey/&#8217;. I’d rather not mention how long those two little gotchas took me to work out.

With the chocolatey.config file updated, open PowerShell, change directories to C:\Temp\chocolatey.0.9.8.28 and run the command ‘cpack’. Take the new chocolatey.0.9.8.28.nupkg and copy it to wwwroot. Chocolatey is now ready to install!

One last thing you’ll want to do before moving on to the Boxstarter packages is set your NuGet install to include the ApiKey for your server (if you didn’t turn it off). Once again, don’t forget the trailing / in the url or you won’t be able to push packages to your repository. Run the command:

nuget.exe SetApiKey chocolateyrocks -source http://SERVER/

Boxstarter Installer

Copy all the Boxstarter folders out of C:\ProgramData\chocolatey\lib and into C:\Temp.

To be honest, I’m not a fan of the current Boxstarter install path of C:\User\User\AppData\Temp\Roaming, so I went through each Boxstarter’s setup.ps1 and changed $boxstarterPath to use $env:programdata instead of $env:AppData. If you don’t care about this change, go ahead and skip it, the most important thing is the config file that must be changed in boxstarter.chocolatey.

Speaking of which, open up C:\Temp\boxstarter.chocolatey.2.4.123\tools\boxstarter.config and modify it as follows:
<ChocolateyPackage>http://SERVER/Packages</ChocolateyPackage&gt;
<ChocolateyRepo>http://SERVER/install.ps1</ChocolateyRepo&gt;
<NugetSources>http://SERVER/nuget</NugetSources&gt;

Then add a copy of boxstarter.config to C:\Temp\boxstarter.chocolatey.2.4.123\tools\Boxstarter.Chocolatey\Boxstarter.zip, overwriting the one already inside.

You can use pretty generic nuspec files in each folder and cpack them all to create the nupkg files. The only one to pay special attention to is the base Boxstarter’s nuspec file, where the dependencies must be set in a specific order.
<dependencies>
<dependency id=”BoxStarter.Common” version=”2.4.123″ />
<dependency id=”BoxStarter.WinConfig” version=”2.4.123″ />
<dependency id=”BoxStarter.Bootstrapper” version=”2.4.123″ />
<dependency id=”BoxStarter.Chocolatey” version=”2.4.123″ />
</dependencies>

As you use cpack in each directory, the last step is to push them to the repository with the following command:

cpush package_name.nupkg -source http://SERVER/

Once everything is in place, you can now to go a client and install everything.

iex ((new-object net.webclient).DownloadString('http://SERVER/install.ps1'))
cinst Boxstarter

Now granted, doing things this way means you can’t use the coolest feature of Boxstarter; Launch from the web.  However, it is still an amazing framework to help configure a system quickly.

Posted in DevOps, Enterprise Management | Tagged , , , , , | 1 Comment

Bake your own Chocolatey NuGet repository

INTRO: I recently attended SoCal Code Camp to check out talks on some DevOps tools I’ve been looking into adding to my toolkit (Vagrant and either Chef or Ansible). The talk on Chef was in the second hour and looked like it would end up being standing room only, so in the first hour I went to the talk that was in the same room to make sure I’d have a seat for Chef. The talk was “Quickly spin up a new windows machine and get your software installed using Chocolatey” by Justin James. I quickly Googled Chocolatey; being a PowerShell guy I figured it sounded pretty cool and I’d be able to save a seat for Chef, win-win. Little did I know that Chocolatey would be the best thing I’d discover at Code Camp that weekend. 🙂

Also, I need to give credit to itToby for his article Setup Your Own Chocoloatey/NuGet Repository.  It was a huge help in getting a jump start in understanding what was going on under the hood with the NuGet Server setup after seeing the demo at Code Camp.  I used that walk-through when I built my first server, but then I kept thinking back to the demo and felt it was much easier…  Thats when I remembered the Chocolatey.Server package that Justin used and figured I could Boxstarter the whole thing.

I can’t think of a better way to show off how cool Chocolatey and Boxstarter.org are than using them to build themselves.  So, get yourself a Windows 2012 server, open up CMD.EXE and run the following:  START http://boxstarter.org/package/nr/url?https://raw.githubusercontent.com/RichHopkins/chocolatey-server-build/master/chocolatey.server.build.txt

START

START

Click through a few Boxstarter prompts and kick back.  Its that easy!

Warning 1

Warning 1

Warning 2

Warning 2

Finished

Done!

Next post…  re-working the Chocolatey and Boxstarter installers and packaging them on your own repository for a fully internal build setup.

Posted in DevOps, Enterprise Management | Tagged , , , , , , , | 4 Comments