The Morning Scrub

Introducing Cronus

2009-09-03T11:44:00.001-04:00

Over the last few years I have been assembling the skills needed to create a “Core” framework that all my future projects should be based on. This framework is a library containing features specifically designed to eliminate the constant need to recreate the wheel every time a new project is started. Let’s face it everyone has this problem. Each time a new project is developed; there is a need for certain functionality like a security model, configuration model, notification system, and navigation system. This is where Cronus comes in.

The creation of this framework is among other things intended to save development time, for this reason I have chosen to name the project after the Greek God Cronus aka the god of human time.

In my next post I discuss the advantages/disadvantages to using Cronus 1.0.

Building A Better “Where” Clause

2009-08-30T16:07:00.003-04:00

Over my career I have learned a lot about SQL Server, unfortunately, sometimes some fundamental ideas get forgotten. This weekend I went to the Jacksonville Code Camp 2009 where I sat in on a session titled “building a better where clause” here are a few of my notes in case anyone forgot of never knew them.

- Try to use indexed fields whenever possible. Searching indexes in much, much faster than un-indexed fields. This means if you are not searching on the primary or foreign key, consider whether the field in questions should either be searched in the first place. If it does, consider adding an index on that field.

- When writing your where clause, take the order into account. Place the expressions most likely to return false first. Take a look at the below queries. Both return the same results, however unless people with the last name “smith” make up 50% of the table the first expression would evaluate faster than the second.

SELECT fname, lname, gender FROM People where lname = ‘smith’ AND gender = ‘male’
SELECT fname, lname, gender FROM People where gender = ‘male’ AND lname = ‘smith’

This is because as the system evaluates each row, deciding if it should be included in the result set, it will move on to the next row as soon as part of the where clause evaluates to false.

- Avoid wrapping table columns listed in the where clause in a function, doing so prevents SQL Server from using any indexes, which means a significant drop in performance.

- Avoid wrapping table columns with upper() or lower() in the where clause unless you SQL server is set to be case sensitive. By default SQL Server is not case sensitive, meaning the functions are not needed. As noted below using these functions prevent the use of indexes.

- Avoid using any form of “NOT”, including not null if possible as all forms of not also prevent indexes from being used.

- When using an in() statement, if possible place the most likely to be found first. Once the comparison evaluates to true the system can immediately move on to the next record.

Hudson, More than Just a River in New York….

2009-08-22T11:01:00.002-04:00

In the past I have written at length about the value of implementing a few select agile principals. One of these principals, “Continuous Integration” (CI) provides so much “bang for your buck”, it just can’t be ignored.

For so long I have wanted to implement continuous integration practices. Unfortunately, until now I never was able to get it going. I did some research on “Cruise Control.net” but all I read pointed to a long and drawn out installation and configuration process. Given the fact that this was a one man show and I would likely not be able to obtain the sys admin support it became less likely that this would be an option. Another issue was the requirement of a server with IIS, meaning I could not simply install it on my workstation for initial testing. Needless to say this was a deal breaker. I looked into another product “TeamCity”, which sounded easier to get up and running but, again it required IIS.

With this I started trying to implement my CI plans via hook scripts attached to our SVN server. These scripts though appeared to work were slow and seemed like a house of cards ready. The idea was to add a post-commit script to automatically checkout new work. The script next called on NAnt to build the project. NANT would also call on FxCop (for static code analysis) and NUnit (for the eventual unit tests we would have). Again, this worked, kind of, but I now felt comfortable sending it into the wild. After all this the project got shelved for a time.

Then one fateful day over lunch I read an article featured in CODE magazine titled “Hudson Continuous Integration Server” by Eric Anderson. Hudson is a continuous integration tool written in Java. As a committed .Net enthusiast I must say I was skeptical. I must say I was pretty lucky to find such a jewel. Within an hour of reading the article I was up and running. I won’t go into details of specific confirmation steps; Mr. Anderson does this so well. Instead I will discuss my impressions of the application and the overall contribution it has made.
Since the app is Java, you must have a recent edition of the JVM. As far as I can tell this is real requirement. I simply put in a few command line statements, and the server installed was installed on my workstation. The whole thing is packaged up into a Java .jar file meaning everything needed is included in this file, upon execution it unpacks and install into memory, no permanent installation is required.

A few more statements and the system was alive and available for configuration via web browser (there is a built in web server). The URL was http://localhost:8080/. The article used a repo from Google Code, with that and of course visual studio I was ready to go. I copied the configuration examples in the article and did a few tests.
Soon I had it linked to a few departmental repositories and now anytime anyone checked in code Hudson would check out a copy and attempt to compile reporting the results. This is a huge advancement. Until now team members checked in code whenever, not knowing that they had forgotten to add one file or another to the repo. Now within a minute Hudson checks out the new changes and attempts to compile them. After I got the system installed on one of our servers, Hudson was be able to include email support, notifying the team of each good or bad build.

Before this system was in place people would check stuff in, then others might check more stuff in, finally at some point someone did an update and their build was broken, stopping work. What followed was usually someone stomping around saying that this system sucks. After that we would attempt to identify the team member who had the missing files and get them added to the repo. Knowing immediately if you broke the build will reduce these incidents significantly, in most cases the team member who forgot a file will know and fix it before anyone else in the team has a problem. Our team needed something better, and here it is.
During my research of “ CruiseControl.net” I became aware of a really nice satellite application enter CCTray. CCTray is a small windows app that sits in the system tray and pulls a feed from the CI server for all projects a specific user subscribes to; it then shows red, yellow, or green depending on the state of the build. The CODE magazine article outlined the basics of what needed to be done to use CCTray with Husdon and again, after a little tinkering we were up and running.

Now that we have been running Hudson for well over a month, I have increased the scope of its responsibilities considerably. I use it to generate change reports for me monthly reports, to analyze and report on adherence to coding standards via its FxCop, and report on compiler warnings reported by MSBuild.
I am totally satisfied with this product and encourage anyone else who wants to implement CI on a shoestring to give it a look.

What is unit testing and why should we do it?

2009-02-14T23:40:00.002-05:00

Throughout the lifecycle of any project, many types of testing should and are done. Usability testing is needed to check if the user interface is easy to use and understand. Security testing is essential for software which processes confidential data and to prevent system intrusion by hackers. Performance of load testing checks to see if the software can handle large quantities of data or users. This is generally referred to as software scalability. Each of these types performs an important role. One type of testing that is often overlooked it known as unit testing.

Unit testing is the practice using small bits of code to exercise code that you have written. In many ways it is something that we have all been doing all along. Every time you single out a specific method and test it intensely to make sure it works as expected you are in a way unit testing your code. This is ok, but wouldn’t it be nice if you could re-execute these tests over and over again throughout the lifetime of the project to ensure that the classes and methods you worked so hard to write continue to work as designed even as other aspects of the project are added or modified? Too often we as a team, write something, test it, and later someone else is directed to make some other change, that causes previously coded features to un-expectantly fail. This is when unit testing comes in.

Instead of manually testing each component, visually inspecting the results, we can use a testing framework to exercise the code and evaluate if it still works or not. This works because we simply write a test that emulates the manual test that would have been written. For example, if you had written a math class and needed to test the add method, you could call that method passing 2 and 3 outputting the results. If it returned 5 it would seem reasonable to say that the add method works. Instead you could create a small test method that would also call the add method it would look something like this:

public void TestAdd()
{
Assert.AreEqual(5,MyMathClass.Add(2,3));
}

This method says the your add method should return 5 if it is passed 2 and 3 if it does not the test would be reported as a failure. Now that the test is save for all to re-run at any time. As each developer writes more components, they contribute more tests, which all would be run over and over again.

There are many excuses not to test
Though I think few would even attempt to argue that testing in general is not needed, most times much of it is left to the end of the project or even skipped completely. In-fact many times it is even regarded as a nuisance, causing more work than it is worth. I have heard countless excuses why testing is considered a second class citizen. My favorite is that we are too busy and simply don’t have time to do any more than minimal testing. I personally, am shocked every time I hear these words spoken. Consider for a moment how much time has been spent debugging code that was believed to be stable, just to find out later that it is not. How much time has been spent trying to track down bugs reported by users that cannot be easily replicated? In the end the cost of the test at the end strategy is far more than implementing a pay as you go approach.

One analogy I recently read said that it is far easier to mow the yard each week, then to try and cut a virtual forest once a week. Think about this for a moment, if more attention was paid to testing as the application was developed, many of the bugs would have been found and fixed before the system became too complex and too many other components were build atop the flawed logic.

Coding with Confidence
Invariably, by the time each project is enters the maintained phase, the team has already begun to distrusts the code base, and many already look forward to the day it is retired. As features are changed, developers do the absolute minimum to meet the new requirements. This means a lot of legacy code is left in the project for no other reason than the team member is terrified that removing or refactoring anymore than is absolutely necessary will lead to cascade failures across the system. This is where unit testing frameworks are so powerful. Tests that had been previously written can be run at anytime, ensuring each atomic element continues to perform as designed. This allows developers to code with confidence, instead of coping with the paralysis that so often causes us to do only the bare minimum.

The next step
So where do we start? Now that we hopefully agree in principal that unit testing is a good thing, we need a road map to implementation. The first step is to adapt a testing framework from which we can build all of our tests.

There are quite a few frameworks available the leaders in the .Net community include Microsoft Test, MBUnit, and NUnit among others.

Microsoft’s product though it does offer integration with Visual Studio. It first became available in Visual Studio 2005 Team Foundation, and limited support in now available in Visual Studio 2008 professional, but there are many features that remain unavailable to us. Additionally, this product has generally been regarded as an immature product by many prominent members of the .Net Community.

That leaves MBUnit and NUnit both are variations of the popular XUnit family of testing suites derived from Java’s JUnit.

MBUnit is the newer of the two frameworks; it actually builds upon NUnit, and has some interesting new features. Unfortunately, it is not nearly as widely used as NUnit and has far less available documentation available. It is for that reason I have chosen to us NUnit. It has been around for over 10 years, and the amount of information and support for this product is virtually endless. The only thing that remains is to get started with our first test…

(NUnit has an addin called TestDriven.Net which allows for tighter Visual Studio integration)
NUnit is probably the most mature unit testing framework of the bunch. It also has the biggest adoption and community support. There are more articles, discussions, tools and add ons for NUnit than any of the other frameworks combined. Just look at this link on google trends that compares searches for each of the frameworks. No other even comes close.

What is Continuous Integration and Why Do We Need It II

2009-01-23T09:51:00.001-05:00

So now that we have a very general understanding of what CI is the next step is to take a look at and see how it can work for us. On projects where sources control is used, we can have the system attempt to build before code is actually accepted into the repository. This will seriously cut down on the issues we had recently with someone forgetting to include a file, and failing the build for the next person who comes along. This is a good start but we can do better.

Over the last few months I have written and sometimes pulled articles about different utilities that can help us increase departmental productivity. Now it is time to take advantage of a few of them.

Lets Review:
FxCop
“FxCop is a free static code analysis tool from Microsoft that checks .NET managed code assemblies for conformance to Microsoft's .NET Framework Design Guidelines.” In short, this is an application that looks at the compiled code and verifies proper usage of coding standards set by Microsoft standards. This includes unused variables, misspellings in code, naming conventions and tons of others. Unfortunately, someone has to run it at specific intervals, generate a report and see to it that the changes are made, at that point it has to be run again, and the cycle starts over again. Imagine sending something back and forth over and over, that’s not to mention there is no way to identify the developer responsible for the offending code.

NUnit
NUnit is a popular testing framework for Microsoft .Net. Its main function is to allow developers to develop unit tests for their application.

A unit is the smallest testable part of an application. In object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class. A unit test is designed specifically to make sure a single unit is functioning as expected. Once developed, these unit tests could be run each time a change is made, thus identifying any new bugs that were introduced. Unfortunately, like these test only work if they are run and sometimes people forget to do this before checking their code in.

NAnt
NAnt is a free and open source software tool for automating software build processes. NANT executes its build process based on an XML build file. It can intern call command line instances of NUnit, FxCop, NDoc, and other tools during the build process. Configurations can be made so that if a unit test fails or a standard is not me the build is failed.

NAnt can also handle automated deployment, If the build is successful, it will generate a release build, zip it and place it in the releases folder for later deployment to the production server. All of this is kicked off by one command line call.

This is good, however, some still has to execute NAnt to build and verify their code before checking it in. This is where CI servers such as CruiseControl.Net and other come in. They bind to the source control product, such as Subversion, and on check-in download a fresh version of the code. From here they execute NAnt (including FxCop, NUnit, and others) if anything goes wrong the team receives an email and the code is rejected. The developer fixes whatever issue was reported and then the code is accepted. The better news is all these utilities are open source and free. Even better, I have already done most of the work to integrate everything together. The only thing I don’t have it the integration server mentioned above, because that will require some further assistance from the network team.

What is Continuous Integration and Why Do We Need It?

2009-01-23T09:50:00.000-05:00

Continuous Integration is a set of software engineering practices that speed up the delivery of software by decreasing integration times.

Sounds good but what are these practices?
Although these practices existed before, extreme programming practitioners recommend that teams require their developers to continuously integrate. I will give more detail about the practices later on, for now here is the list:
· Maintain a code repository
· Automate the build
· Make your build self-testing
· Everyone commits every day
· Every commit (to mainline) should be built
· Keep the build fast
· The build needs to be fast, so that if there is a problem with integration, it is quickly identified.
· Test in a clone of the production environment
· Make it easy to get the latest deliverables
· Everyone can see the results of the latest build
· Automate Deployment

Maintain a code repository
I think everyone is on board with the concept of a code repository. It provides versioning of the source code add allows management to see who did what when through comments placed in the revision log each time code is checked in. Consider for a moment, how many times code has been checked in, only to later cause someone else to download code that wont compile? How much time was spent trying to figure out what the problem was and resolve it?

Automate the build
Enter the automated build, this means that the entire system should be buildable using a single command. This is in contrast to a manual build process where a person has to perform multiple, often tedious and error prone tasks. The goal of this automation is to create a one-step process for turning source code into a working system. This is done to save time and to reduce errors.

Each time the system is built the code is compiled, if for any reason the build fails (including syntax errors or missing resources) everything stops. This code should not be accepted into the repository.

Make your build self-testing
The system should be written with tests that verify that it performs as intended. These tests should be run as part of the build. The best way to make the build self-testing is to create “unit tests” for each granular function within the business layer. At build time each of these unit tests could be run to ensure that previously written functionality still works and is compatible with the latest updates. To get the most from unit testing a strict barrier between the UI, business, and data layer should be implemented.

The value of a self testing build grows exponentially as your application gets older and more complex.

Everyone commits every day
By committing regularly, every committer can reduce the number of conflicting changes. Checking in week’s worth of work runs the risk of conflicting with other features and can be very difficult to solve. Early, small conflicts in an area of the system cause team members to communicate about the change they are making.

Every commit (to mainline) should be built
Commits to the current working version should be built to verify they have been integrated correctly. A common practice is to use Automated Continuous Integration. For many, continuous integration is synonymous with using Automated Continuous Integration where a continuous integration server or daemon monitors the version control system for changes, then automatically runs the build process.

Keep the build fast
The build process needs to be fast, this way if a problem with integration is identified, it is quickly identified and addressed by the appropriate team members.

Test in a clone of the production environment
Having a test environment can lead to failures in tested systems when they are deployed to the production environment, because the production environment may differ from the test environment in a significant way.

Everyone can see the results of the latest build
It should be easy to find out whether the build is broken and who made the change. This provides a valuable “state of the application” while at the same time providing a measure of accountability.

Automate Deployment
Automated deployment removes the human factor or “Magic” from deployment, as the system always follows the same procedures and therefore is less error prone.

Source Control

2009-01-13T20:03:00.000-05:00

For some time now I have been very excited about the benefits offered by source control (SC). It provides a clear and precise record of changes made to code along with who made them and when good comments are added then it can say why.

Through our recent .Net projects we have become more familiar with one specific system for source control, Subversion (SVN) along with its popular client TortoiseSVN. I would agree that there have been ups and downs, however, looking back I think we are much better off having used then if we had not. Recently, a new client application became available, Ankh. This app is not exactly a replacement for Tortoise, instead more of a companion. The most important service that Ankh offers is that it integrates directly in to Visual Studio. Though I have not yet had time to play around with Ankh, It sounds very exciting.

Having enjoyed all the benefits of source control on our .Net projects for more than a year now, I have been very interested in using it with other languages our department develops with, most notably ColdFusion. Unfortunately, it doesn’t seem to be conducive to the way our CF apps are developed. Though each developer working on a .Net project runs a copy locally when developing, our network administrators policy is not to install CF express on each machine. This means that all CF source code resides on the server and is edited from there, with developers sharing a single copy. Though I really would like to use source control I have yet to divise a practical approach to doing so, therefore, all CF source code remains outside of source control.

Another area that would be nice to have under SC is the database. I have read many articles explaining that schema should be kept in SC, I have yet to read a comprehensive approach that would also cover data, particularly configuration data that is just as important as the schema itself. Without this data under SC then it seems like using it at all would be pointless.

One idea that I recently had was to ban all config data from being kept in the database, instead it would be kept in xml config files that could be kept under SC.

Recently, I read some interesting thoughts on the matter. Though I have not had time to read all the content, it is what lead me to the xml config file idea… If this interests you please read the 5 part series available at http://www.codinghorror.com/blog/archives/001050.html

A Project and a Professional Renaissance II

2009-01-12T16:45:00.000-05:00

Over the lifetime of any project, there are many different issues that must be resolved in order to successfully complete the project. A few include Staffing, Technology and of course may favorite scope creep. I think now would be a good time to talk a little about some of the staffing issues that I faced during the project.

One of these issues started when we unfortunately, we soon found that we were a little understaffed for a project of this size, so we began by hiring four new developers nearly doubling the size of the programming team. Eventually we parted ways with all of them as 3 of the four had grossly overstated their skills and we had not done much in the way of testing them since these were to be junior positions anyway.

The Three Amigos
The first three, (all held degrees from the college of business MIS specifically) spent weeks developing “hello world” and getting daily instruction from me. We started with the definitions of words like object, class, method and such and then moved on to challenging things like creating our own add and subtract methods within a test class. We worked mostly with console apps, unfortunately, the only things they learned were the things I pounded into them, no one asked any questions, and by no means did anyone do any learning on their own time. This is despite my repeated attempts to mentor them, countless discussions about how exciting programming is, and offering the rest of the team to them so that they could commiserate and hopefully pick up a thing or two.

Eventually the last of the three amigos left standing told me he was fearful that he would lose his job. I saw this as another opportunity to mentor him, so I approached it like a big brother. I asked him why he felt this way. His response was something like “because I am not very good at what I do.” Now it should be noted that he was correct on that point, but I had always thought that I could will the skills on to them, that if I tried hard enough it would click. It came to me easily, I figured at some point it would come to him as well. So my advice to him was that if he felt he and his family were unsecure he needed to take action ASAP. I told him that if he really wanted to be a professional programmer, then he needed worked hard and improved his skills. That he should pick one of the countless technologies that interested him and work to become an expert at it. I noted that he was very lucky because there is an unending supply of information and free training available for the taking (magazines,books, white papers, free lectures, code camps, and online videos to name just a few). I suggested that if he put in the work, then he would probably be able to sleep at night, doing so would undoubtedly improve his standing within the department and if he did lose his job with us, he would have the skills to get and keep the next position. This is advice that I live by; I make it my passion to learn as much as I possibly can thus ensuring my financial security. That next position may not be a dream job but I know there is something out there if need be, and thus I sleep at night. These are views I share with everyone that will listen, success doesn’t just come, you have to earn it.

Sadly, three weeks later when I followed up on the conversation, I asked what choices he had made, what type of studying he had done. The third amigo’s response was none, that books were too expensive and he didn’t want to borrow from anyone else. The next day, I wished him luck and fired him. He had been with us six months and I am not sure he retained any knowledge from the entire time technical or professional.

The Student
The fourth and final new hire was a computer science student. I was very skeptical of what technical skills a student could possibly have at first. As it turns out, I should have been more concerned about his professionalism. This was the last of the four to be hired and really the only one that contributed in any significant way to the project.

When I met him in my office on his first day of work, it quickly became apparent that he had the goods from a coding prospective. I would characterize him as a technology puritan; he was very letter of the law when it came to techniques and concepts. Though I did feel that we as a group needed a little more of that, it did make it difficult at times to get him to do what was needed. Much of our work centers on compromise. I wish I had the luxury of being as unbending as he, but I just would not be able to exist here or probably anywhere else. I think he was only able to operate this way because he is a student and had no real experience working in a production shop. The other item that did cause some friction between us was his lack of professionalism. He often came in hours late without a call or even notifying me when he arrived, left early, sometimes he never did show. True he was a student but I was wooed by his sheer skill and needed the extra set of hands so I often overlooked that fact to my own determent.
Eventually, “the student” found another job. Unfortunatly, he resigned via email.

What did I learn from all of this? I think it helped me grow as a leader, even though these guys floundered, I can only think it is because they really didn’t try; if they had they should have realized some measure of success regardless of my leadership skills. By interviewing, coaching, and eventually firing three out of four of them, I gained critical experiences that may have taken years to acquire otherwise.

A Project and a Professional Renaissance

2009-01-06T18:04:00.000-05:00

A few months back I was extremely involved in a major development project; this project was an effort that distributed nearly $250k between different contractors for different services. Our job was to actually develop the application. This was an ASP.Net project that had been started by another shop several years earlier. We were to implement several hundred pages of improvements and while we were at it, we were directed to bring it up to par with the current industry technologies.

This project which I will code name Pegasus (partly because I like the neat code names from MS and partly because of where I work) was written in .Net 1.1, using stored procedures, and an immensely cryptic Data Access Layer (DAL). It was by no means complete. In fact there was very little that was done by any since of the word. I would call it a prototype, but really it was more of a proof of concept. Still there was a large code base, which we were charged with adapting vs. replacing.

My role was twofold; first I was to serve as team lead, sitting between the “executive branch” where decisions were being made about requirements and deadlines and the “programming arm” where the actual construction was taking place. My other role was an obvious and familiar one, software engineer.

I thought I’d take a little time and document what I learned throughout this long project and comment on where I find myself after its completion, so am going to write a multi part series documenting the goings on (not necessarily chronologically) over the last year or so.

Stay tuned, next time I will discuss staffing complications that arose over the lifetime of the project (at length).

More Tools: NAnt

2008-12-03T16:09:00.004-05:00

NAnt Automated building of .net projects made easy...

Though I have been aware of it for quite sometime, I recently read an article in CoDe magazine the provided detailed instructions as to installation and a sample implementation of NAnt. I had wanted to try it out but was a little turned off by the very dry and difficult documentation I had read up till now. Unfortunatly, this is often the case with techie, open source projects.

First I setup my hello world application, next I followed the instructions and within an hour or so I had a batch file and a build.xml file from which i could execute an NAnt build. The instructions in the CoDe article, Even provided instructions to create either a debug or a release build. They did not stop there however, they also included help in creating a deployment .zip file with everything needed.

Very nice...

The article can be found at the following URL:
http://www.code-magazine.com/Article.aspx?quickid=0811051

What a Tool III

2008-11-18T16:18:00.005-05:00

Introduction to GhostDoc

Published: 30 Oct 2006
By: Roland Weigelt

GhostDoc is a free add-in for Visual Studio that helps developers writing XML documentation comments.

Introduction
The XML documentation comments of C# reward you in multiple ways: Visual Studio shows their content on various occasions (e.g. in tooltips in the text editor, or in the object browser), and some tools (e.g. NDoc or Microsoft's documentation tool code-named Sandcastle) can generate nice looking help files for you. So overall, writing XML documentation comments sounds like a pretty good idea - unfortunately it often consists of a lot of stupid work that simply has to be done.

For example the documentation of constructors ("Initializes a new instance of ."), or properties (how often did you write "Gets or sets ..."? Or worse, "Gets or sets a value indicating whether..."?). Often enough you need the same documentation over and over again, e.g. when working with interfaces or class hierarchies.

What does GhostDoc do?
GhostDoc installs a new command for Visual Studio's C# source code editor. When editing a source file, you simply navigate the cursor into the method or property you want to document, call that command by pressing a hotkey (or using the context menu) and GhostDoc will insert an XML documentation comment. Much like typing "///" in front of the method or property, but instead of creating just an empty skeleton, GhostDoc tries to generate large parts of the actual documentation.

If your class member is an interface implementation or an override of a base class member, GhostDoc will use the existing documentation, regardless of where the interface or base class comes from. So basically what this means that you can re-use tons of documentation that Microsoft wrote - just remember the last time you've implemented IEnumerable and thought about how to comment the GetEnumerator() method.

In those cases where no existing documentation can be used, GhostDoc tries to guess the texts for the comments. While this idea seems to be a bit strange at first, under certain conditions (we'll get to that later) GhostDoc actually does a pretty good job. Often enough the results are not accurate or even downright funny, but on average correcting the mistakes/problems in the generated documentation takes less time than writing the whole comment from scratch.

But how can a tool with virtually no understanding of the English language generate often pretty accurate documentation? Well, the basic idea is pretty simple: GhostDoc assumes that your code follows the Microsoft's Design Guidelines for Class Library Developers:

you are using PascalCasing or camelCasing to write identifier names consisting of multiple words
your method names usually start with a verb
you do not use abbreviations in identifier names

If you follow these rules (e.g. write "ClearCache()" instead of "Clrcch()") and choose more or less self-documenting identifier names, there's a good chance for GhostDoc to split the identifier names into words, tweak and/or shuffle them a little and produce a comment that, while not perfect, gives you a head start on the way to good documentation.

The text generation is performed using customizable rules and templates, and in addition to the built-in rules it is possible to define new custom rules that extend or replace existing rules (by giving the custom rules a higher priority and/or disabling built-in rules)

As mentioned above, GhostDoc does not really understand English identifier names, nevertheless it tries to apply certain mechanisms to increase the quality of the generated comments:·

Handling of verbs (GhostDoc assumes the first word of a method name to be a verb): Add -> Adds , Do -> Does, Specify -> Specifies.
"Of the" reordering (using trigger words like "width", "height", "name", etc.): ColumnWidth -> Width of the column
Combined with special handling of specific adjectives: MaximumColumnWidth -> "Maximum width of the column" instead of "Width of the maximum column"
Automatic detection of acronyms consisting of consonants (Html -> HTML), combined with a list-based approach for other acronyms (Gui -> GUI)
Use of a list of words where that should not be headed by a "the" (AddItem -> Adds the item, but BuildFromScratch -> Builds from scratch)

As the quality of the comment depends heavily on the quality of the identifier name, using GhostDoc for a longer period of time actually teaches writing consistent and self-documenting identifier names, which is definitely a nice side-effect.

What doesn't GhostDoc do?
GhostDoc cannot perform miracles, and the way it guesses documentation may not work well with your personal style. And GhostDoc is not a tool for creating the complete documentation for a given source file in one go. You can only document one member at a time - which is by design and fully intentional, because you always have to check and, if necessary, correct the results of each text generation.

Summary
GhostDoc is a free add-in for Visual Studio that automatically generates XML documentation comments. Either by using existing documentation inherited from base classes or implemented interfaces, or by deducing comments from name and type of e.g. methods, properties or parameters.

References
GhostDoc can be downloaded from the GhostDoc website:http://www.roland-weigelt.de/ghostdoc

(This article was taken from the DotNetSlackers website)

What a Tool II

2008-11-07T15:25:00.003-05:00

Source Monitor 2
Type: Management
Source: Campwood Software
Download URL: http://www.campwoodsw.com/sourcemonitor.html
Price: free

The freeware program SourceMonitor lets you see inside your software source code to find out how much code you have and to identify the relative complexity of your modules. For example, you can use SourceMonitor to identify the code that is most likely to contain defects and thus warrants formal review. SourceMonitor, written in C++, runs through your code at high speed, typically at least 10,000 lines of code per second. SourceMonitor provides the following:

Collects metrics in a fast, single pass through source files.
Measures metrics for source code written in C++, C, C#, VB.NET, Java, Delphi, Visual Basic (VB6) or HTML.
Includes method and function level metrics for C++, C, C#, VB.NET, Java, and Delphi.
Saves metrics in checkpoints for comparison during software development projects.
Displays and prints metrics in tables and charts.
Operates within a standard Windows GUI or inside your scripts using XML command files. Exports metrics to XML or CSV (comma-separated-value) files for further processing with other tools.

(This article was taken from the Campwood Software website)

What a Tool!

2008-10-27T08:00:00.001-04:00

So, lately I have been interested in learning more about the tools that are out there to help me increase the productivity of our rather small team. After a little searching I stumbled onto http://www.visualstudiogallery.com/ , this was a great source for new and interesting tools to support .NET development. As I Scanned through the offerings I saw there were toys for everyone. I thought that since I had a budget of little or none there would be just about that much offered to me. I was sure wrong about that… There are Code Metrics and Code analysis tools to help me ensure we have as high quality a product as possible, there are documenting tools for the members of the development team, there are even a few toys for the LINQ enthusiasts among us.

I am going to break my evaluation of these tools in to the above suggested 3 categories. First Management, then tools to assist in everyday development, and last things that were just neat toys but not all that useful.

FxCop
Type: Management
Source: Microsoft
Download URL: http://msdn.microsoft.com/en-us/library/bb429476(VS.80).aspx
Price: free

FxCop is actually the utility that started this whole adventure, I recently read about as a sidebar in an article in one of the many .net developer oriented magazines. I did a quick search and found it available from our good friends in Redmond. It holds itself out as a rules based code analysis tool. Out of the box it comes preloaded with rules near and dear to the folks at Microsoft, it included more of their standards than I ever knew existed.

After downloading it and kicking the tires a bit I proceeded to turn it loose on our departments most recently deployed application. In no time it had a few suggestions to offer. By a few I mean 5000, and when I say suggestions, that is what they actually were. The application offered suggestions as to how to bring each warning into compliance. It reported on unused variables, anything that did not meet the .Net guidelines for naming, tense, even using a dictionary to make sure variables had a name that made since.

For my money, this is a great application to ensure that everyone working on the project stays on track, I plan to run it at regular intervals ensuring no other application gets so far out of spec, because in the end, we all want code that is as standardized as possible; it is just easier to support.

Next to be evaluated: Source Monitor 2

What is LINQ

2008-10-24T08:00:00.002-04:00

LINQ (Language Integrated Query) is a component released within the .NET 3.5 Framework. It is one of the most powerful features of .NET 3.5. It serves the purpose of querying objects. It is a Microsoft programming model and methodology that essentially adds formal query capabilities into Microsoft .NET-based programming languages. LINQ offers a compact, expressive, and intelligible syntax for manipulating data. The real value of LINQ comes from its ability to apply the same query to an SQL database, a DataSet, an array of objects in memory and to many other types of data as well.
There is no one definition for LINQ, but it aims to solve the problem where we currently use different techniques for manipulating and selecting data from databases versus XML versus object collections; LINQ aims to make our life easier by giving us a coordinated, consistent and efficient syntax from our development environment and by using your one chosen programming language, rather than switching between programming languages.The current LINQ family of technologies and concepts allows an extensible set of operators that work over objects, SQL data and XML data sources. The generalized architecture of the technology also allows the LINQ concepts to be expanded to almost any data domain or technology, so what LINQ is today will expand in the future.

LINQ to SQL
The LINQ to SQL provider is just one part of the overall framework, however, it is a very powerful and attractive option when and ORM (see Object-relational mapping below) is called for. LINQ uses an SQL-like syntax to make query expressions well beyond the capabilities of embedded SQL as implemented in programming languages. That's because embedded SQL uses a simplified, streamlined syntax to add SQL statements to other programming languages, where there's no attempt to integrate such statements into the native syntax and typing mechanisms. Thus, you can't invoke native language structures such as functions in embedded SQL statements, as you can using LINQ, because it is implemented to use native syntax, structures, and typing mechanisms. Furthermore, LINQ may be used to access all kinds of data, whereas embedded SQL is limited to addressing only databases that can handle SQL queries.

In short LINQ to SQL can operate as middle ware between business objects and the database, the advantage of this may seem abstract but consider the following example. In SVT if we want to get just a single standard and do some work on it we first have to create a new object of type standard, next we tell the new object that it’s id is say 21. From here the standard object has been programmed to take that id and go call on other objects from the data namespace, it call a specific one the passes the standard id to a stored procedure which returns a query containing the desired info. Next, that query is returned to the standard object and it essentially, loops over all the columns in the query and populates a property in the new standard object for each one. Now (very simplistically speaking) we have our standard object, it has the methods and properties needed to do work. After we change a few things and tell the object to update the database, the whole process happens again in reverse. If on the other hand we were using LINQ we would simply write a 1 line SQL like expression passing the id of the standard we desired, and it would return a standard object ready to go. We could then edit that object and tell it to update and be done with it.

LINQ to Objects
The LINQ to Objects provider is used for querying in-memory collections, using the local query execution engine of LINQ. The code generated by this provider refer the implementations of the standard query operators as defined in the Sequence class and allows IEnumerable collections to be queried locally.

What this means is that when we want specific information from an array or other type of collection, instead of looping over the entire array, building another array of the desired data, we can simple query the array as if it were a table in a database.

Conclusion
The overall gain in productivity we realized by switching to LINQ is immeasurable, instead of taking all day to get everything needed in the SQLGateway class up and running, our team can do a LINQ query in seconds returning full blown already initialized objects. In an effort to make the implementation as effective as possible I dedicated months of personal research, read many books on the subject, countless articles, and attended several workshops. I felt that to make it all work I needed to become the departmental expert helping the others along as they were ready. As each team member came on board with the new concepts they too became very excited with the prospects and really wish other languages such as Coldfusion had similar features.

Object-relational mapping (aka ORM, O/RM, and O/R mapping) is a programming technique for converting data between incompatible type systems in relational databases and object-oriented programming languages. This creates, in effect, a "virtual object database" which can be used from within the programming language.

Coming out of your Shell Part I

2008-07-27T15:20:00.001-04:00

Over my career as a software engineer, I have learned that no matter what the requirements are, most web apps are made up of the same basic components. A few of these components include a site layout, menu system, configuration, user management, a security model and a data model. For some time now I have been interested in developing a “shell” application from which most of my new applications could be derived. This shell application would need to include these “core” components, and would serve as the foundation, from which the rest of the app could be constructed.
I have considered using a portal system such as Xoops (PHP) of .Net Nuke (VB). Both do meet the above described requirements, including user management and providing extensibility points known as modules. Unfortunately, I feel that the core is designed with the needs of the open source community in mind, not those of my clients. Additionally, as it is an “off the shelf” product, I doubt that I would be able to effectively understand the complexity and extend the core as needed.
It is my intention that this open ended series will document my quest to develop “my” very own “shell application”, one that can be adapted and continually enhanced. It will provide both a starting point and eventually I hope an iron clad core for all future web apps.

Some of C# 3.0's new features

2008-05-26T20:27:00.001-04:00

C# 3.0 provides many new technologies, recently I became aware of some of the syntax improvements. This post is intended to outline some of the highlights.

Auto Implemented Properties
One new feature in C# 3.0 is “Auto Implemented Properties”, this is a fancy way of saying that we no longer have to create the private property and the public fields when we write new classes. Take a look at the below example:

private string userId;
Public String UserId
{
get { return userId; }
set { userId = value; }
}

IS the same as

public string UserId {get;set;}

The compiler will write out the needed getters and setters, however, they need not be in the source code unless a more complex structure is needed. The net effect is 5 less lines of code per property.

Object initializers
Another new feature is known as “Object Initializers”, this is a means to allow programmers to set values that aren’t part of the constructor at init time. Take a look at the below example:
Customer c = new Customer(“john”,”doe”);
c.phone=”123-456-7890”;
c.zip = “43534”;

is the same as

Customer c = new Customer(“john”,”doe”) {phone=”123-456-7890”,zip=”43534}

This may seem insignifigant, however, it is not. There are many classes I have come in contact with that have properties that aren’t set in the constructor. In these cases you must make additional statements to set the further needed properties. That limitation is now removed, you can now set those properties all at once. This again, it a little compiler trickery, at compile time they are broken out into individual statements just as you would have, but as with the auto implemented properties, this is an improvement in the overall readability, serviceability, and cleanliness of the source code. I a way this reminds me of how you can add items into an array at the time of declaration
string[] x = new string[]{“a”,”b”,”c”};

Local Variable Type Inference
Microsoft has seen the complaints that the following just looks a little odd:
Customer customer = new Customer();
Their answer is called local variable type inference using the var keyword. What this means that the following will now be accepted:
var customer = new Customer();
In this case the compiler detects the type from the new Customer() part of the statement. I cant say this is a whole lot shorter or neater in most case, however, when using linq, the type is not known as types are created on the fly to support the data requested. So it does seem to be useful in the linq arena.

More thoughts as they come in.

LINQ to SQL Overview" video

2008-05-20T21:32:00.005-04:00

From TechEd Barcelona,
This presentation from Luca Bolongnesa is heavy on the Italian accent, but also heavy on content. This is probably the best video, podcast, or article I have read to date on Linq to SQL. Luca covers

how the Linq to SQL engine works, explaining in depth scenarios, and demonstrating the sql that was executed.
How joins and aggregation are accomplished
Stored Procedures
Optimistic Concurrency and Pessimistic Concurrency
Transactions both distributed and non distributed

System.Collections.Generics - Dictionary

2008-05-19T13:55:00.004-04:00

So, I have wanted to dive into the world of generics for some time now. I have read and read about the stack, queue, and the dictionary classes but just didnt have a need until now. recently on project foo, the need to use an "anchor" that would never change and get back the current display text was presented. I considered my options, we could query for each value... No, there would be many many instances of this need on the same page, and though stored procs are fast, I felt that would be clumsy. Then a light came on, use the dictionary object.

I tried to take it a step farther and use a linq query and the todictionary extension method, to create my dictionary, unfortunatly this went nowhere. I was able to write the needed statement for the linq query, but had many problems with the todictionary method. I suspect it is partly because i am new to both generics and linq. I found that there were few good examples of how to use the todictionary method that I could reference, so I ended up doing it the old fashon way, creating a dataset, then creating a dictionary, and finally itterating over the dataset/datatable and adding each record.

LINQ to SQL and Datacontexts

2008-05-18T10:21:00.007-04:00

At first I thought LINQ to SQL was going to be as simple as creating a datacontext, adding all tables to it and then linq away. After some work and some research, I have discovered that though that did work a little more thought is required. For one, what happens if the database schema changes, would the datacontext be updated? Second, should I have a global all encompasing datacontext or would there be a performance cost?

Most of the how-to's and chapters in books i have read do not really cover this, they simply explain how to setup a new datacontext and then how to begin "linqing". I did a little research and discovered that the data context does not get updated when you make changes to the database schema.

So what is the right way to do this, what should go where, and when? One article i found on the subject can be found at http://www.west-wind.com/weblog/posts/246222.aspx . This article by Rick Strahl a Microsoft MVP, suggests the a global datacontext be avoided. After reading several articles on the matter I am inclined to go with a datacontext per thread. This relitively easy to implement, and circumvents the issue he discusses with the global data context forcing all users to updates at one time. I still, havent found any discussion as to weather the whole database should be modeled in a single context or if seperate contexts should be created to hold logical groups of tables. Additionally, I have yet to see any real discussion on the issue that happens if the database schema changes.

There is a code camp coming up on 6/7/2007 and I plan to follow the linq track exclusively.

Refractor SQLGateway Class to Linq

2008-05-17T15:37:00.002-04:00

Currently the SQLGateway class is the most confusing and difficult file in the entire system to manage. Unfortunately, it is also the one files that every other file must interact with as it is just as the name states, the gateway to the database. I have held off on this for some time as I wanted to make sure I understood as much as I possibly could about what was being done and what could be done instead.
Here are my conclusions:
1. This class was developed with the intention of abstracting the data layer from the business layer. This is a fantastic idea, in that it can provide a means for reusing queries, and provide a central point of change should the system ever need to be ported over to another DBMS such as Oracle. This follows the spirit of what we as programmers have been taught to do, but were never able to accomplish since the very beginning.
2. Though the idea of providing a single point of change should we use another DBMS is here, it would be impossible in practice to actually accomplish the needed changes to 6100 lines of code along with the other supporting classes, interfaces, and stored procs in any reasonable amount of time.
3. This highly overblown abstraction makes it extremely difficult and time consuming to add new features such as new objects/tables or even new fields as there are about 10 places that need to be updated for each,
4. When the system was initially developed there was obviously a decision made that dictated that all forms of CRUD (insert, update, delete, and select) would be implemented for all data objects. I find this to be a security risk, but beyond that it is very unnecessary. Perhaps the SQLGateway class was generated, and not to be edited by hand, it sure seems so.
5. These standards and techniques were developed to support a .net 1.1 application and thus are subject to review as many years have passed and the app is now under .net 3.5
6. .net 3.5 offers a new technology known as linq or language integrated query. This new data access methodology, provides a 1-3 line interface to the database through ORM (object relational mapping). This interface also provides a parameterized approach which is the number 1 reason for using stored procs, as if additional programming is needed ala a stored proc it still offers these as methods od the data object it creates. Behind the scenes linq creates a data class to mode each entity/table in the database, thus providing the abstraction called for in the 3 tier environment.

I plan to use linq instead of and in place of the current data model wherever possible, Since this application is already quite developed, meaning there is already a large system in place, and the team and myself are all a little new to linq, I plan to implement a 4 tier environment. 1 UI layer (already provided), 1 business layer ( already provided), and 2 data layers stacked. The first will be the existing SQLGateway class and its supporting cast. The second data layer will be interacted with through the first. This means all the existing data methods in the SQLGateway class will remain, and will return what they always have, however inside them, as time permits, I will adjust them to use linq instead of the traditional stored proc setup. This change, along with removing unneeded methods to update, insert, and delete system table data will in my calculations reduce the size of the SQLGateway class by 50%, increase productivity among our team, and still offer every bit of the value of the old model.

First Entry

2008-05-17T15:31:00.001-04:00

This blog will be a way of documenting and colaborating within the department about what we have and are still learning in the .net world.

As this blog is dedicated to Asp.net I thought I'd start off by mentioning, if you havent already been there you should surf http://www.asp.net/ This is a fantastic site for information, and has great free how-to videos (direct link) http://www.asp.net/learn/. These videos are better than those from learnvisualstudio.net in my opinion.

Happy Learning,

-Lendog