What is Continuous Integration and Why Do We Need It II

So now that we have a very general understanding of what CI is the next step is to take a look at and see how it can work for us. On projects where sources control is used, we can have the system attempt to build before code is actually accepted into the repository. This will seriously cut down on the issues we had recently with someone forgetting to include a file, and failing the build for the next person who comes along. This is a good start but we can do better.

Over the last few months I have written and sometimes pulled articles about different utilities that can help us increase departmental productivity. Now it is time to take advantage of a few of them.

Lets Review:
FxCop
“FxCop is a free static code analysis tool from Microsoft that checks .NET managed code assemblies for conformance to Microsoft's .NET Framework Design Guidelines.” In short, this is an application that looks at the compiled code and verifies proper usage of coding standards set by Microsoft standards. This includes unused variables, misspellings in code, naming conventions and tons of others. Unfortunately, someone has to run it at specific intervals, generate a report and see to it that the changes are made, at that point it has to be run again, and the cycle starts over again. Imagine sending something back and forth over and over, that’s not to mention there is no way to identify the developer responsible for the offending code.

NUnit
NUnit is a popular testing framework for Microsoft .Net. Its main function is to allow developers to develop unit tests for their application.

A unit is the smallest testable part of an application. In object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class. A unit test is designed specifically to make sure a single unit is functioning as expected. Once developed, these unit tests could be run each time a change is made, thus identifying any new bugs that were introduced. Unfortunately, like these test only work if they are run and sometimes people forget to do this before checking their code in.

NAnt
NAnt is a free and open source software tool for automating software build processes. NANT executes its build process based on an XML build file. It can intern call command line instances of NUnit, FxCop, NDoc, and other tools during the build process. Configurations can be made so that if a unit test fails or a standard is not me the build is failed.

NAnt can also handle automated deployment, If the build is successful, it will generate a release build, zip it and place it in the releases folder for later deployment to the production server. All of this is kicked off by one command line call.

This is good, however, some still has to execute NAnt to build and verify their code before checking it in. This is where CI servers such as CruiseControl.Net and other come in. They bind to the source control product, such as Subversion, and on check-in download a fresh version of the code. From here they execute NAnt (including FxCop, NUnit, and others) if anything goes wrong the team receives an email and the code is rejected. The developer fixes whatever issue was reported and then the code is accepted. The better news is all these utilities are open source and free. Even better, I have already done most of the work to integrate everything together. The only thing I don’t have it the integration server mentioned above, because that will require some further assistance from the network team.

What is Continuous Integration and Why Do We Need It?

Continuous Integration is a set of software engineering practices that speed up the delivery of software by decreasing integration times.

Sounds good but what are these practices?
Although these practices existed before, extreme programming practitioners recommend that teams require their developers to continuously integrate. I will give more detail about the practices later on, for now here is the list:
· Maintain a code repository
· Automate the build
· Make your build self-testing
· Everyone commits every day
· Every commit (to mainline) should be built
· Keep the build fast
· The build needs to be fast, so that if there is a problem with integration, it is quickly identified.
· Test in a clone of the production environment
· Make it easy to get the latest deliverables
· Everyone can see the results of the latest build
· Automate Deployment

Maintain a code repository
I think everyone is on board with the concept of a code repository. It provides versioning of the source code add allows management to see who did what when through comments placed in the revision log each time code is checked in. Consider for a moment, how many times code has been checked in, only to later cause someone else to download code that wont compile? How much time was spent trying to figure out what the problem was and resolve it?

Automate the build
Enter the automated build, this means that the entire system should be buildable using a single command. This is in contrast to a manual build process where a person has to perform multiple, often tedious and error prone tasks. The goal of this automation is to create a one-step process for turning source code into a working system. This is done to save time and to reduce errors.

Each time the system is built the code is compiled, if for any reason the build fails (including syntax errors or missing resources) everything stops. This code should not be accepted into the repository.

Make your build self-testing
The system should be written with tests that verify that it performs as intended. These tests should be run as part of the build. The best way to make the build self-testing is to create “unit tests” for each granular function within the business layer. At build time each of these unit tests could be run to ensure that previously written functionality still works and is compatible with the latest updates. To get the most from unit testing a strict barrier between the UI, business, and data layer should be implemented.

The value of a self testing build grows exponentially as your application gets older and more complex.


Everyone commits every day
By committing regularly, every committer can reduce the number of conflicting changes. Checking in week’s worth of work runs the risk of conflicting with other features and can be very difficult to solve. Early, small conflicts in an area of the system cause team members to communicate about the change they are making.

Every commit (to mainline) should be built
Commits to the current working version should be built to verify they have been integrated correctly. A common practice is to use Automated Continuous Integration. For many, continuous integration is synonymous with using Automated Continuous Integration where a continuous integration server or daemon monitors the version control system for changes, then automatically runs the build process.

Keep the build fast
The build process needs to be fast, this way if a problem with integration is identified, it is quickly identified and addressed by the appropriate team members.

Test in a clone of the production environment
Having a test environment can lead to failures in tested systems when they are deployed to the production environment, because the production environment may differ from the test environment in a significant way.

Everyone can see the results of the latest build
It should be easy to find out whether the build is broken and who made the change. This provides a valuable “state of the application” while at the same time providing a measure of accountability.

Automate Deployment
Automated deployment removes the human factor or “Magic” from deployment, as the system always follows the same procedures and therefore is less error prone.

Source Control

For some time now I have been very excited about the benefits offered by source control (SC). It provides a clear and precise record of changes made to code along with who made them and when good comments are added then it can say why.

Through our recent .Net projects we have become more familiar with one specific system for source control, Subversion (SVN) along with its popular client TortoiseSVN. I would agree that there have been ups and downs, however, looking back I think we are much better off having used then if we had not. Recently, a new client application became available, Ankh. This app is not exactly a replacement for Tortoise, instead more of a companion. The most important service that Ankh offers is that it integrates directly in to Visual Studio. Though I have not yet had time to play around with Ankh, It sounds very exciting.

Having enjoyed all the benefits of source control on our .Net projects for more than a year now, I have been very interested in using it with other languages our department develops with, most notably ColdFusion. Unfortunately, it doesn’t seem to be conducive to the way our CF apps are developed. Though each developer working on a .Net project runs a copy locally when developing, our network administrators policy is not to install CF express on each machine. This means that all CF source code resides on the server and is edited from there, with developers sharing a single copy. Though I really would like to use source control I have yet to divise a practical approach to doing so, therefore, all CF source code remains outside of source control.

Another area that would be nice to have under SC is the database. I have read many articles explaining that schema should be kept in SC, I have yet to read a comprehensive approach that would also cover data, particularly configuration data that is just as important as the schema itself. Without this data under SC then it seems like using it at all would be pointless.

One idea that I recently had was to ban all config data from being kept in the database, instead it would be kept in xml config files that could be kept under SC.

Recently, I read some interesting thoughts on the matter. Though I have not had time to read all the content, it is what lead me to the xml config file idea… If this interests you please read the 5 part series available at http://www.codinghorror.com/blog/archives/001050.html

A Project and a Professional Renaissance II

Over the lifetime of any project, there are many different issues that must be resolved in order to successfully complete the project. A few include Staffing, Technology and of course may favorite scope creep. I think now would be a good time to talk a little about some of the staffing issues that I faced during the project.


One of these issues started when we unfortunately, we soon found that we were a little understaffed for a project of this size, so we began by hiring four new developers nearly doubling the size of the programming team. Eventually we parted ways with all of them as 3 of the four had grossly overstated their skills and we had not done much in the way of testing them since these were to be junior positions anyway.


The Three Amigos
The first three, (all held degrees from the college of business MIS specifically) spent weeks developing “hello world” and getting daily instruction from me. We started with the definitions of words like object, class, method and such and then moved on to challenging things like creating our own add and subtract methods within a test class. We worked mostly with console apps, unfortunately, the only things they learned were the things I pounded into them, no one asked any questions, and by no means did anyone do any learning on their own time. This is despite my repeated attempts to mentor them, countless discussions about how exciting programming is, and offering the rest of the team to them so that they could commiserate and hopefully pick up a thing or two.


Eventually the last of the three amigos left standing told me he was fearful that he would lose his job. I saw this as another opportunity to mentor him, so I approached it like a big brother. I asked him why he felt this way. His response was something like “because I am not very good at what I do.” Now it should be noted that he was correct on that point, but I had always thought that I could will the skills on to them, that if I tried hard enough it would click. It came to me easily, I figured at some point it would come to him as well. So my advice to him was that if he felt he and his family were unsecure he needed to take action ASAP. I told him that if he really wanted to be a professional programmer, then he needed worked hard and improved his skills. That he should pick one of the countless technologies that interested him and work to become an expert at it. I noted that he was very lucky because there is an unending supply of information and free training available for the taking (magazines,books, white papers, free lectures, code camps, and online videos to name just a few). I suggested that if he put in the work, then he would probably be able to sleep at night, doing so would undoubtedly improve his standing within the department and if he did lose his job with us, he would have the skills to get and keep the next position. This is advice that I live by; I make it my passion to learn as much as I possibly can thus ensuring my financial security. That next position may not be a dream job but I know there is something out there if need be, and thus I sleep at night. These are views I share with everyone that will listen, success doesn’t just come, you have to earn it.


Sadly, three weeks later when I followed up on the conversation, I asked what choices he had made, what type of studying he had done. The third amigo’s response was none, that books were too expensive and he didn’t want to borrow from anyone else. The next day, I wished him luck and fired him. He had been with us six months and I am not sure he retained any knowledge from the entire time technical or professional.


The Student
The fourth and final new hire was a computer science student. I was very skeptical of what technical skills a student could possibly have at first. As it turns out, I should have been more concerned about his professionalism. This was the last of the four to be hired and really the only one that contributed in any significant way to the project.


When I met him in my office on his first day of work, it quickly became apparent that he had the goods from a coding prospective. I would characterize him as a technology puritan; he was very letter of the law when it came to techniques and concepts. Though I did feel that we as a group needed a little more of that, it did make it difficult at times to get him to do what was needed. Much of our work centers on compromise. I wish I had the luxury of being as unbending as he, but I just would not be able to exist here or probably anywhere else. I think he was only able to operate this way because he is a student and had no real experience working in a production shop. The other item that did cause some friction between us was his lack of professionalism. He often came in hours late without a call or even notifying me when he arrived, left early, sometimes he never did show. True he was a student but I was wooed by his sheer skill and needed the extra set of hands so I often overlooked that fact to my own determent.
Eventually, “the student” found another job. Unfortunatly, he resigned via email.


What did I learn from all of this? I think it helped me grow as a leader, even though these guys floundered, I can only think it is because they really didn’t try; if they had they should have realized some measure of success regardless of my leadership skills. By interviewing, coaching, and eventually firing three out of four of them, I gained critical experiences that may have taken years to acquire otherwise.

A Project and a Professional Renaissance

A few months back I was extremely involved in a major development project; this project was an effort that distributed nearly $250k between different contractors for different services. Our job was to actually develop the application. This was an ASP.Net project that had been started by another shop several years earlier. We were to implement several hundred pages of improvements and while we were at it, we were directed to bring it up to par with the current industry technologies.

This project which I will code name Pegasus (partly because I like the neat code names from MS and partly because of where I work) was written in .Net 1.1, using stored procedures, and an immensely cryptic Data Access Layer (DAL). It was by no means complete. In fact there was very little that was done by any since of the word. I would call it a prototype, but really it was more of a proof of concept. Still there was a large code base, which we were charged with adapting vs. replacing.

My role was twofold; first I was to serve as team lead, sitting between the “executive branch” where decisions were being made about requirements and deadlines and the “programming arm” where the actual construction was taking place. My other role was an obvious and familiar one, software engineer.

I thought I’d take a little time and document what I learned throughout this long project and comment on where I find myself after its completion, so am going to write a multi part series documenting the goings on (not necessarily chronologically) over the last year or so.

Stay tuned, next time I will discuss staffing complications that arose over the lifetime of the project (at length).