GAT
Guidance Automation related entries.
Software factories is an elusive concept nowadays. In this article, I will try to help you understand which technologies can be used today to build production-quality factories and the reasons for the perceived inconsistency in the tools and the vision on Microsoft's part.
None of what follows is speculation: I've been a main developer on one of the two core technologies that realize today' software factories (patterns & practices Guidance Automation Toolkit or GAT), and as such I've been in close contact with the teams and architects building the technologies I will explain here (including Jack Greenfield, Wojtek Kozaczynski, Mauro Regio and others). Also, my company is a leader in software factory development, providing most of the people that build the factories coming out of the most active practitioner in the area today: the Microsoft patterns & practices group. We also have several customers around the world that we helped reap the amazing productivity boost that today's factory tools can bring.
First and foremost: this article is NOT about the grand vision (still underway, rest assured, but nevertheless in the realm of the future) outlined in The Book: Software Factories: Assembling Applications with Patterns, Models, Frameworks, and Tools by Jack Greenfield, Keith Short et all. I will talk about technologies that are ready for production today. This also means that the strict concept itself of a SF outlined in the book needs to relaxed for what's currently possible with the available tools. You can think of today's factories as infants that will grow with features as times goes by and the end-to-end SF vision becomes a reality. That doesn't prevent the "infant" from giving you a big boost in terms of productivity and software quality.
You can build SF today by using a combination of the following two tools:
-
DSL Tools: or Domain Specific Language Tools. This is a toolkit for creating VS-integrated graphical notations for a given model that you basically make up. It allows you to construct a custom-tailored designer for a model for your problem domain.
-
GAT/GAX: or Guidance Automation Toolkit/Extensions. This is a toolkit for extending VS with solution, project and item templates, code snippets and so-called recipes (or developer use cases) exposed as context menus within VS, that guide the developer in completing a complex task.
The problem is that integrating the two is not a trivial task. Why you may ask? Aren't they both coming from Microsoft? And how does GAT/GAX fit in the SF vision? After all it's not even mentioned in The Book!
Glad you asked ;). In order to understand this "mess", and figure out how to make it work for you, you need to understand a little bit of the background behind the two technologies. The DSL Tools is basically the first step in the road to realizing the full SF vision outlined in The Book. Hence, if you try to build a real-world factory just using DSLs, you will find it fairly incomplete and not supporting typical scenarios such as providing an initial solution structure where you will put your DSLs, etc. The missing features, though, are most probably those that will come (or become unnecessary) as the full vision is implemented. In the meantime, it feels lacking.
On the other hand, Microsoft patterns & practices (p&p) is NOT about building a compelling but somewhat distant feature, but about solving customers' problems today. p&p follows a process called "Customer Connected Engineering", which means customers get to influence what we do. Delivering guidance on how to best use the platform has been p&p traditional focus, but as guidance became more and more complex, thicker books and more complex libraries (think CAB and EntLib) just didn't cut it anymore. Customers were not getting the guidance in an easy to consume way. Hence, we set out to find a way to bring the guidance into Visual Studio, in a way that was easy to author (not only for p&p but for other companies delivering guidance internally) and contextual. The readily available VS extensibility mechanisms (add-ins and VSIP packages) did not satisfy either requirement, so we built our own framework on top of VSIP to allow for easier extensibility. And GAX was born. As the name implies, it's a mechanism for automating the guidance that you would have to otherwise perform manually by following a book's instructions.
You may notice that p&p software factories do not contain DSLs. This is in part because the goals of p&p SF are somewhat different than The Book SF. Their main purpose is to guide the developer through the best practices for implementing a given solution. In the end, it's the developer himself who implements the solution. It just so happens that he does so much faster and in a predictable way because he uses automated guidance within VS that generates code, unfolds boilerplate projects and solutions, provides code snippets, etc.
Given the different origins and goals of the two technologies, it may seem that they are hopelessly irreconcilable. Fortunately, that's not necessarily the case, and there are many ways in which they can be integrated. My partner Victor Garcia Aprea and Mauro Regio have published an article on MSDN about integration scenarios that include integrating the two.
A number of integration points are particularly critical:
- An initial solution structure is generally required: this can be done with GAX.
- Typically at a specific location in the solution, a domain model/designer could be added to design some area of the application (i.e. a business entities domain model, that generates business entities + data transfer objects + data access layer code + SQL files with stored procs for the DB). This needs to be done with a combination of DSL and GAX if you want the DSL to only be available at a specific location in the solution. (remember, GAX is about context)
- With the DSL in place, you may want to generate multiple code files (as in the case of the business entities model I mentioned above), maybe even in different projects, and maybe you even need to generate the projects themselves. Again, DSL tools alone is not enough. By default, it will do two things that are pretty bad for a real-world factory:
- The code (a T4 template) that transforms the domain model in a designer into actual code for your project is expected to live in the same project as the domain model file. You are given a custom tool that runs the transformation, and a solution explorer toolbar icon to run all the transformations in the solution/project ("Transform All Templates"). What kind of guidance are you giving your developers if they can modify it at will just by modifying the T4 template that generates their code? It's more like giving them an example that they can tweak, and typically you don't want your developers modifying the templates for code generation that your architects together with your senior devs have created to ensure quality code.
- Each T4 template results in exactly one big file with the output of running the codegen against the domain model. If your model designs business entities, you will end up with a huge file with all the entities in the model. Generally not a good idea, and certainly goes against the typical .NET coding practice of using one file per class.
And it will not do the other two things that you need: generate new projects, and generate files that target multiple projects. You can supplement the DSL with GAX in order to achieve all of this. Achieving the above two goals requires deep understanding of how both technologies work at a very granular level, though, but it's definitely possible (more on this later).
- You may want to launch recipes/execute actions from specific shapes in the domain model diagram (i.e. "Configure Security for Entity" or something like that, maybe even regenerate code for a specific model entity and not all of them). You can hook GAX recipes to the context menus on DSL diagrams pretty easily, and with some more work you can make it contextual enough to provide very granular visibility (i.e. only model entities that have property Foo = true).
Of course, the integration is not as seamless as it could have been. But I hope I shed some light on the current situation by sharing the background and how the two tools came to be. Moving forward, you can expect the experience to become much better, as the scenarios that led to GAT/GAX are integrated into the SF toolkit.
As you might have guessed, we (Clarius) have gone over each and every hop in this integration that you can possibly imagine. We have recently released a CTP of our Software Factories Toolkit, which helps integrate the two technologies, and which you can download for free.
And probably you know it is based on the super-cool Guidance Automation Extensions (GAX) and Toolkit (GAT). Read
Tom's post about what's new and improved in this release.
So you're hooked to this cool GuidanceAutomationToolkit. Now, say you have a cool recipe you'd like to place in an arbitrary context menu, such as the context menu over a Web Reference. In VS any menu is called a command bar. Now that particular context menu is not one of the command bars that GAT provides friendly aliases for:
Fortunately, GAT allows you to specify Guid and ID attributes that uniquely identify any menu in VS, so you can place your recipe anywhere you want. That's great, you say, but how in the world are you gonna know what values to use there? Turns out that until last week, I was fishing myself for those numbers in the VS SDK .ctc files (you don't even want to know what those are, believe me). Turns out that there's a much better and easy way, so I'd like to share it here.
The trick involves customizing the menu you want to put your recipe on, and then comparing the exported VS settings with a previous unmodified version, so you can tell the diff. The steps are:
- Go to Tools > Import and Export Settings...
- Select
Export selected Environment Settings option.
- From the tree, unckeck the root and then select only the
All Settings\General Settings\Menu and Command Bar Customizations node.
- Click Next. Ensure the message displayed says the export was successful and without errors (IMPORTANT!).
If you get an error, close and reopen VS, and try again without any solution opened. If you keep getting an error, I'm afraid you may need to reset your settings to the defaults (in the same dialog, select Reset all settings instead of Export...
- Open a project where you have a Web Reference. Select the Web Reference in the solution explorer
- Right-click on the toolbar and select
Customize
- On the
Toolbars tab, mark the Context Menus. At this point, a new toolbar with a number of dropdown menus should have appeared. Those are all the context menus currently available.
- Expand
Project and Solution Context Menus. You will see a number of entries. Now it's just a matter of finding the context menu you're looking for. In the case of the web reference, it's a little tricky (and that's why I chose this example) as the one you have to look for is named Folder (yuck!). But in general the names are pretty intuitive.
- Now we need to introduce a customization in this menu so that it shows up in the exported settings later on. So make a trivial change inside it, such as renaming one of its child items.
- Close the
Customize window.
- Repeat steps 2 to 4 to export the settings again, with a different file name this time.
The safest way to do the process above without risking your current settings is to use a virtual machine and reset all settings before doing the export. For the cleanest possible .settings file, it's also recommented that you reset to General Development profile.
Now open a diff program and compare the two files. You should look into the <UserCustomizations> node, and you should see a <modify> element with the change you made. In my case, I got:
<modify Cmd="{1496A755-94DE-11D0-8C3F-00C04FC2AAE2}:0000045c"
CmdPri="03000000"
Group="{D309F791-903F-11D0-9EFC-00A0C911004F}:000002a3"
GroupPri="02000000"
Menu="{D309F791-903F-11D0-9EFC-00A0C911004F}:00000431"
Name="Updates Web Reference"/>
Note that I changed the name of the Server Explorer item to Server Explorers. You can then rollback by simply importing the original settings you exported. The important piece is in bold above: the Menu attribute. That value is the Guid and ID you're looking for! Keep in mind that the ID after the ":" is an hexadecimal value, so you have to convert it to an integer.
From the information above, I can now place a new recipe in the View menu just by specifying:
<CommandBar Guid="D309F791-903F-11D0-9EFC-00A0C911004F" ID="1073" />
Here's the proof ;-)
Now, the most interesting thing is that you don't have to do *anything* at all in order for that menu to be hidden whenever the associated recipe is not available (i.e. it's associated with an unbound recipe reference that specifies that it should only be enabled on certain targets, or it's a bound recipe associated with a particular solution element). In cases like the above, when you're already putting the recipe in a very specific context menu, that may not be such a compelling feature, but if you're placing your recipe in a common menu (like some of the main menus), it's a good feature to have. It makes for a less cluttered user experience.
What follows are some thoughts regarding the authoring of guidance that we have learned during practical experience. The following applies to both DSL and GAX toolkits.
Developing guidance should be an iterative process, and we're still exploring it and how it fits in the overall development process. Intuitively and based on previous experience building several guidance packages, I'd say the process is more or less as follows:
Phase 1: Define End Product
1. Use intensive TDD and short iteration cycles to develop the end "product" you'd like to guidance-enable (i.e. code, application structure and architecture, etc.)
2. During 1), you will end up with a process that takes you from a scenario/use case to an implementation using the architecture/structure you designed (ideally via TDD)
Phase 2: Define Guidance Process and Flow
3. From that process, figure out which are the variability points, where the user should be involved in decisions that affect what the code/solution looks like, as well as the dependencies between them (user does A and only then can do B)
4. Understand and clarify the roles, personas, concerns and use cases the guidance should express
5. Based on the previous two findings, define the launch points for those user interactions (recipe/template/dsl launching points)
6. Mock-up the recipes and UI required for the entire package, and document the steps and input required for each
7. Analyze the mocks, go through them, and evaluate whether the input information, the process flow, and the launch points follow a natural progression that is likely to be intuitive enough for users. Also, think of potential missing pieces of input that may be needed to get to the end result from there
Phase 3: Implement Guidance
8. Finally, add the actions to the recipes so that they generate the code
a) Optionally, the recipes may generate tests that exercise the features in the end result (code/application). I say this is optional, because this is a process that should have been previously exercised and sufficiently proven during steps 1 and 2. Generating from the same recipe, code and its tests is not TDD at all, and if steps 1 and 2 are well done, may be a waste of time, as you already know that the code you will emit will adhere to the architecture and design principles outlined there.
9. Test the generated code and the whole process. Once this is OK (or if you have spare time in the meantime), do the next step.
10. Improve the UI by adding type converters and UI editors
Unless you do Phase 1, the optional 8.a step becomes more important, but I think it is far worse as a testing approach, and will never be as comprehensive as a well done Phase 1.
Phase 1 and 3 are the most likely to consume the most time. I believe in the majority of cases (unless you already know very well you want the guidance to do up-front, or the scope well defined and not too big), the former will be longer than the latter. However, depending on the nature of the tooling part, and the complexity/lack of documentation/unforeseen bugs-issues with VS/integration with Beta-quality products/etc., Phase 3 may become a big part of the work too (we faced this a number of times, where something was supposed to take a few hours and we ended up spending at least a couple days chasing a bug or erratic behavior in VS). Any kind of integration with VS is bound to be unpredictable to a certain extent in that front for the foreseeable future, I'm afraid :(
What raises considerable the bar for phase 3 is if you introduce a meta-guidance in the picture (i.e. you want a guidance package to help people build guidance packages in a certain area).
These are roughly the things that you need to change in your current package:
- Search replace "50505" with "51206" in the entire solution. This is the changed assembly version number that appears in all your .vstemplates.
- Delete all references from all projects to Microsoft.Practices.* and re-add them pointing to the new version of the binaries.
- Search replace "igt" with "gax" (for XML and .vstemplate files)
- All T3 templates need to be updated to the T4 syntax shared with the DSL tools
- Search & replace: Microsoft.Practices.RecipeFramework.Library.Actions.T3Action, Microsoft.Practices.RecipeFramework.Library with Microsoft.Practices.RecipeFramework.VisualStudio.Library.Templates.TextTemplateAction, Microsoft.Practices.RecipeFramework.VisualStudio.Library
- DteHelper class now lives in the Microsoft.Practices.RecipeFramework.Library namespace and assembly.
- In .vstemplate files, you could previously specify initial state for your recipes with the following syntax:
<TemplateReference Name="Projects\ServiceAgent\ServiceAgent.vstemplate" Target="\ServiceAgents">
<XmlSerializableHashtable xmlns="http://schemas.microsoft.com/pag/igt-xmlhashtable" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<Entries>
<Entry>
<Key xsi:type="xs:string">Key1</Key>
<Value xsi:type="xs:string">Value1</Value>
</Entry>
</Entries>
</XmlSerializableHashtable>
</InitialState>-->
</TemplateReference>
That has been replaced by the much simpler format:
<TemplateReference Name="Items\SerializableClass.vstemplate" Target="/" >
<InitialState xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<Entry>
<Key>SerializableField</Key>
<Value>Name</Value>
</Entry>
</InitialState>
</TemplateReference>
Of course, you can still use xsi:type just like the XmlSerializer does to get strongly typed initial values. You can put pretty much whatever you want there, as long as the value is compatible with the corresponding arguments defined for the recipe.
I think that's all, but let me know if you find anything that's missing or not working!
The Guidance Automation Extensions (GAX) and Toolkit (GAT) have been updated for Visual Studio 2005 RTM. As Tom describes it:
At long last, the Guidance Automation Toolkit (GAT) and Guidance Automation Extensions (GAX) have been updated for the final version of Visual Studio 2005. For those that came in late, GAT allows developers and architects to integrate guidance deliverables such as blocks, frameworks and patterns into the Visual Studio environment, using mechanisms such as templates, wizards and code generation. GAT is targeted at people authoring these "guidance packages", while GAX is the runtime component required by anyone consuming guidance packages. You'll be seeing quite a few p&p deliverables built using GAT (and hence requiring GAX) over the next year, so this would be a great time to get up to speed on this exciting technology.
Just as a quick list of things that GAX/GAT enables you to do, *very* easily:
- Create reusable actions that automate VS using either the DTE or the VSIP interfaces.
- Create rich argument collection strategies for use in wizards using .NET standard TypeConverter and UITypeEditor base classes. This also means that you get the built-in ones supported in your wizards for free!!
- Create automatic argument calculation strategies, that can determine (default) argument values from the environment (such as the opened solution, currently selected project properties, VS state, etc.)
- Declaratively design wizards using XML, by specifying the arguments that need to be collected, their .NET type and converters/editors to use to assist the user and validate the input, and even custom pages (a special kind of user control) for advanced input scenarios!
- Group the arguments used by the actions, the automatic argument collection strategies, the wizard to manually them from the user, and the set of actions to perform on the environment in so-called recipes.
- Easily associate recipes to item, project or solution templates, so that additional arguments can be collected for the VS template expansion and parameter replacement, as well as to execute actions after the unfold operation finishes.
- Easily expose those recipes as commands in main or context menus. Assign icons to them simply by pointing to an image file! (no more unmanaged VC++ resources for icons or those ugly .CTC files defining commands in VSIP!!!)
- Create code generation templates using a text templating language code-named T4 that is shared with the DSL Tools (similar in spirit to classic ASP, CodeSmith, NVelocity, etc.)
- Easily contextualize the recipes and templates so that they appear only for:
- A particular element in the solution (i.e. a specific project, item, solution folder, etc.)
- Elements that satisfy a given condition, coded easily in any .NET language (i.e. all XML files, XML files whose root element belongs to a given namespace, all C# projects that are class libraries and have a reference to Indigo, etc.)
- Easily create and package code snippets with your guidance
- Automatically create an MSI setup file to include all the above!
If the above list grabbed your interest, go and download GAX and GAT and start playing with it! Don't forget to ask your questions either on the MSDN forum or the GuidanceAutomation.net community site.
Read the following entry on how to upgrade from Beta2-based GAT packages and the new version.
