imason

Enterprise Search: Ask for help after you have a starting point

Jeff Dunmall — Thu, 11 Feb 2010 20:28:14 GMT

Your employees can't remember all of the information sources inside your company, much less keep track of the search interfaces that exist. Each of the following system likely has its own search interface: SAP for customer data, Ariba for expense data, Siebel for contracts, the E: drive for sales related information, http://www.yourintranet.com to look up product info, unless it is related to the US division, in which case you should visit http://www.us-yourintranet.com. Information about Project...(read more)

Options for a SharePoint 2010 Development Environment

Peter Grigoriou — Sat, 06 Feb 2010 05:41:00 GMT

One of the first things you will probably ask yourself when you start embarking on SharePoint 2010 development is where do you set up your development environment. As I started investigating this, I realized there are a number of various options each with their own implications. Below is listing of each option I found and some of my personal thoughts on each of them:

Option	Pros	Cons
Install SharePoint 2010 directly on our Windows 7 / Vista workstation/laptop	It's nice to have SharePoint available to you at any time.	I don't like the "clutter" it adds to my laptop as I like to keep it fairly clean in terms of what's installed. Pretty manual install
Create a bootable Hyper-V image	Keeps your development environment clean and partitioned from your business applications	If I have to boot to that image, I don't have access to my other application such as Office unless I decide to install all of that on the image which I don't think is very practical.
Create a Hyper-V image on a centrally located Hyper-V Server	Similar to bootable option except you can usually take advantage of better hardware which usually mean better image performance	If you are offline and can't connect to the image there's not much you can do.
Run VMWare Workstation	It supports 64-bit OS's which means you can run it on your Windows 7 or Vista environments without having to install SharePoint 2010 directly on your box. Allows you to keep your image portable	Not a great solution if you want to standardize on Hyper-V which can result in a little pain when you want to share an image.
Run Windows Server 2008 as your workstation/ laptop OS with Hyper-V	Works great if you have the right hardware.	Performance can be horrible if you don't have the right hardware. (Details on this here)

You may have noticed in the list above that there isn't a Microsoft Virtual PC option. That's because it doesn't support 64-bit OS's which is a requirement for SharePoint 2010. I have tried to find out if there is a plan for Microsoft to update Virtual PC to support 64-bit OS's however I have yet to hear anything that indicates this.

For my purposes, I decided to run a VMWare image because I just felt it worked well for me given the hardware I have, the desire to keep my laptop clean, and the fact that I want to remain portable with respect to the image. One other thing to note, if you want to take advantage of the new SharePoint 2010 developer tools in Visual Studio 2010 you'll need to have SharePoint 2010 installed on the same environment as Visual Studio 2010 which, in my opinion, makes the choice of the environment you choose all the more important.

I'd be really interested to hear other's thoughts on this topic.

Some noteworthy new features of SharePoint 2010

Peter Grigoriou — Mon, 07 Dec 2009 03:57:00 GMT

In my last post I mentioned I had the privilege of attending the SharePoint 2010 conference. In this post I'd like to review the things I think were quite notable based on the sessions I attended:

Sandboxed Solutions: I have run into a lot of clients who want to use SharePoint as their application development platform and have various groups within their organization building their own applications. However the question that is always asked is how do we make sure that they can’t do something that brings down the entire farm. Sandboxed solutions provide the ability to limit access to resources and isolate worker processes from the rest of the platform reducing the risk that a particular application will bring down the farm.
Configurable Deployment: Deployment has always been big challenge and resulted in a lot of effort with SharePoint applications. With Visual Studio 2010 there is now a series of out of the box deployment steps as well as the ability to create your own custom deployment package. From what I have seen so far I think this is going to greatly improve SharePoint deployments.
Branch Caching with Windows Server 2008 R2 and Windows 7 and the Office Document Cache in Office 2010: These are features to improve the performance of global SharePoint deployments (ex. A centralized SharePoint Intranet being accessed from around the world.) Office 2010 provides an Office Document Cache which basically stores documents requested for SharePoint locally on the persons computer. Subsequent requests to the same document in SharePoint will result in a check to the server if the document as changed and if not, will serve the document from the local cache and if it has changed will send just the differences over (this is limited to Word, Excel, PowerPoint and OneNote). Branch caching provides the ability to cache documents at particular locations via either a Windows Server 2008 R2 server or on local Windows 7 workstations. Once cached subsequent requests will either go to the local Branch cache server (Windows Server 2008 R2 Box) or request the document from a local peer workstation thus reducing the need to transfer the document over long distances.
Multilingual User Interface: This is the ability to install multiple language packs and toggle between languages on the same site. Prior to this you would need a third party or completely custom solution to make this work and usually involved a lot of development effort. I am not sure of all the limitations but it’s a step in the right direction.
Social Computing: There are a lot of features that pull in Social Computing concepts from Facebook, YouTube, Twitter etc that should add a bunch of excitement to what is possible from a development perspective.
Phonetic People Search: This was pretty impressive. The demo showed the presenter misspelling a person’s name (ex. Cowfman) and it finding the person named Kauffman. It also does the same for common nicknames.

Other notable items:

All Search Web Parts will be open source.
Vastly improved SharePoint Designer Workflows (particularly being able to package them into a WSP so that you can deploy the workflow to other lists and SharePoint sites).
Very scalable farm architecture with a new concept of the Application Services Server. You can view the architecture here.
Integrated SharePoint support in Visual Studio IDE
Improved Search and FAST integration

I'll dive into this in more detail in future posts as I work with some of these items.

Another version of SharePoint? Actually SharePoint 2010 is pretty impressive.

Peter Grigoriou — Tue, 24 Nov 2009 04:47:00 GMT

So this is my first blog post...ever! Yeah, I'm pretty excited that I have finally taken the plunge and decided to write about something, so here goes...

I have been working at imason for over 6 years now and in that time my main technology focus has been around SharePoint. I was lucky enough to attend the SharePoint 2010 conference in Las Vegas and have to say I came away being thoroughly impressed at what I had seen over the 4 days. If you had asked me prior to attending the conference what I thought about SharePoint 2010 I would have replied "Oh great, another version of SharePoint that I need to get up-to-speed on". Today, however, I am pretty excited about the new features and improvements in 2010 and I think it will make for some great potential solutions once it is released.

The SharePoint conference itself was fantastic. The sheer size (around 7500 attendants) and number of sessions (over 250) was very impressive and the sessions I did attend were very informative and went smoothly considering it is a beta product. I think the thing that stuck out to me the most was that for 2010 Microsoft focused on fixing the issues that developers and business users routinely complained about in MOSS 2007, providing a wealth of additional features that really solidifies SharePoint as a great development platform, and building in a lot of the social computing concepts we see in the public domain. Microsoft did a great job of focusing on what was new and improved which helped me understand the key differences between MOSS 2007 and SharePoint 2010. One of the things that can be overwhelming with a product the size of SharePoint is having the knowledge that something exists so that you can leverage it when needed and the conference was great at providing those details. At some point, you will probably be asked the question, "Why should we move to 2010?" and my hope is that over the next few weeks and months my posts will help provide some insight into helping you answer that question.

Performance Testing: The Performance Test Plan

Boyan Tsolov — Wed, 11 Nov 2009 03:57:00 GMT

I have always had a problem doing performance testing (aka load testing). I am quite adept at recording web tests and collecting counters, but I still find it troublesome interpreting the counters that I gather from load tests. The problem is that I need a plan that will produce relevant results with which I can conclude with confidence: "This system is a juggernaught! It can withstand more hits than Google.com." If you are like me and you have trouble planning your load tests, then maybe this post will help you. The following post describes how I like to run my own plan and what conclusions I can make with this plan.

Special thanks goes out to Jeff Dunmall, our co-CEO, whose article "Real-World Load Testing Tips to Avoid Bottlenecks When Your Web App Goes Live" got me to write something more about load testing here, and Randar Puust who uses this plan (and thus now I use it too).

There are a number of facets to load testing:

- setting up a load testing rig and setting up your environment in Visual Studio

- recording the web tests

- planning the load tests

- collecting performance counters and figuring out what the recorded result mean

Unless you have a plan, planning the load tests and reading the results will be meaningless. A performance test plan will keep you on track throughout the last two phases mentioned above. The results from a test plan will allow you to determine:

- maximum number of users that can hit the site before it becomes unresponsive

- can the system recover from overload

- are there memory leaks in the code

A performance test plan consists of four types of tests that need to be run in the following order:

1. Performance Tests

2. Load Tests

3. Stress Tests

4. Endurance Tests

1. Performance Tests

These are the benchmark tests. By this point you should have recorded web tests which cover the most-used parts of your web application. Now, you will create a load test that gathers and runs all of the web tests that you have recorded. Run the load test with settings similar to these:

- load: low end of the expected number of users for the system. I like to start with 5 or 10.

- time: 10 minutes

- think time: 5 seconds

- warmup: 30 seconds to 1 minute is more than enough

A load test like this should not stress the web server, it should be low-impact. The goal of this test is for it to be run later on in this plan and be used to compare against (i.e. a benchmark). Collect and save the result counters from the web server and from the client. Web Server counters to collect are: "CPU %", "Used Memory". Client counters to collect are: "Respose time per page", "Number of Requests per second". Other counters you might like to save are "pages that take the longest time to load", "number of errors", "network usage" - these are useful for neat reports that you might want to compile.

2. Load Tests

The load tests are going to stress the web application and try and find its breaking point. The goal of these test will be to find the lucky number X. This number is the number of users that are concurrently hitting the system and that cause the system to be unresponsive or response times to be unacceptable. For this part of the plan you will need to run a number of load tests. In each case, increase the number of users and collect the counters:

- load: 10 users, 15 users, 20 users, 50 users.... until the response times get too large, or server CPU % gets dangerously high.

- time, think time, warmup: same as for the performance test. Basically we are running the performance test from step 1but with more and more users.

By the end of this step you will know what is the maximum number of users that can hit the system.

3. Stress Tests

Stress tests check that the web application can recover from errors induced by overload.

- load: X + 20% (where X is the magic number from step 2 which causes the system to be unstable)

- time: 1 minute

- think time: 5 seconds

After this test is run the system should become unresponsive. If it is not unresponsive, then increase the 20%. One way to check if it is unresponsive is to check that response times are suddenly exponentially far worse than the results in step 2 and CPU % usage on the server is close to 100%.

Immediately after this test runs run the performance test from step 1. If the performance test runs fine and the results are comparable to the test that was run at step 1, then your system passes! It can safely recover from overload issues.

4. Endurance Tests

The endurance tests are unrelated to the other 3 tests. These tests are run over a long period of time, which causes many users to hit the site. If there are memory leaks, then over the course of this endurance test the free memory on the web server should decrease dramatically.

- load: X - 20% (where X is the magic number from step 2 which causes the system to be unstable). We'll use -(minus) 20% because we don't want to overload the system

- time: 12 hours

- think time: 5 seconds

Remember to record the "Free" or "Used Memory" on the web server. The goal at the end of this step is to make sure that the memory usage on the web server has not decreased dramatically and thus there are no memory leaks in your code. If that is the case with your results then tip of the hat to you!

At the end of these four types of tests you should have a better idea of where your system stands in terms of load that it can handle. You will know:

- what is the maximum number of users it can handle before performance is unacceptable

- can it recover after an overload

- are there any memory leaks

Happy testing!

Create TFS Work Items directly from Outlook Emails

Stephen Kearns — Mon, 19 Oct 2009 14:36:50 GMT

This is a great little productivity tool for teams actively using Work Items to track dev activities in TFS. Used it at a recent customer (thx Dilip!) and found it helpful.

http://blogs.microsoft.co.il/blogs/srlteam/archive/2007/03/04/Team-System-Outlook-2007-Addin-_2D00_-v1.0.aspx

technical; TFS; work items

Don’t Tell Me You’re Not a Designer

Kerri McKenna — Thu, 06 Aug 2009 15:03:00 GMT

Often, when I’m working on a project with someone who isn’t an Interaction Designer I’ll inevitably hear the phrase, “I’m not a designer, but...” followed by an idea or suggestion for changing the user experience. This is frustrating to hear.

Designers are facilitators of ideas. We assemble requirements, best practices and ideas into visual representations for consideration. But we’re not omniscient divas who will storm off at the first hint of a non-designer’s opinion.

Good designers analyze the big picture and the minute details of a design problem. We should be able to tell you why our concepts work well, and where there’s room for flexibility. Rather than telling you that our design is perfect, we need to hear your feedback. We want to collaborate with you to improve our design – it will probably never be perfect, but it can always get better.

So if you’ve got an idea for changing a design or interaction, throw it out there. Sure, designers may have more experience and knowledge about the rules of engagement for design and usability principles – but a good designer won’t shoot you down. Just like we hope that you won’t immediately dismiss our first take on a design solution. Let’s work together here – no ideas are bad, they just might not be the best ones. But we can filter out the good ones together.

Think of a designer’s concepts as sparks for a fire of ideas. I don’t care that you’re not a designer. I do care about your sparks.

Enterprise Search: Why Search Scoping Doesn’t Work

Jeff Dunmall — Tue, 28 Jul 2009 21:57:00 GMT

My previous post outlined the problem with the ‘traditional’ approach to search scoping. To explore the problem and a solution that leverages Microsoft FAST search, I’ve invented a fictitious company called Canadian Financial . That gives us some context – it is a bank with multiple geographies, lines of business, languages, and products. There are dozens of internal sites up that provide a massive amount of information required to do your job more effectively, if you can...(read more)

A tribute to Michael Jackson

Bob Brown — Wed, 15 Jul 2009 20:24:00 GMT

Last night I found a great site commemorating Michael Jackson by letting people post a video of themselves doing the moonwalk. Anyone who knows me, knows that I'm a huge fan, so I took the opportunity to have some fun and shoot a mini clip of me moonwalking to add to the Eternal Moonwalk reel!

If you want to see me, here's the direct link: http://www.eternalmoonwalk.com/bob-nxv. I'm in the clip right after Michael Jackson...

imason in the news with Microsoft Office 2010 Technical Preview Launch Event

Stephen Kearns — Wed, 15 Jul 2009 15:13:00 GMT

Jeff Dunmall, co-CEO and Stephen Kearns, Principal Consultant recently participated as Partners in the Microsoft Office 2010 Technical Preview Launch Event.

First a little terminology – the Technical Preview is an “engineering milestone” indicating that product development has been substantially completed such that Microsoft is ready to start showing it off and generating excitement about the new product. However, it’s not showing up in a shrink-wrapped package just yet.

Jeff, participating from the World-Wide Partner Conference in New Orleans was quoted in IT Business:

“Customers are coming to us and telling us they want to get involved with Office 2010. That's unusual. We normally enter a proactive sales approach with new releases. We have to talk to customers about the benefits and differentiators and that gets the customers engaged. With Office 2010, they're already excited about it,” Dunmall said.
This wasn't the case for Office 2007, Dunmall said. Customers hesitated on Office 2007, but for Office 2010 it has turned into enthusiasm.” [1]

Stephen, participating in Toronto at the Microsoft Canada launch event was quoted in IT World Canada and IT Business respectively:

Microsoft partners at the event said extending the ribbon interface marks a commitment to continuing to include the context-sensitive GUI in future versions of Office. In the words of Stephen Kearns of imason Inc., "It's not going away." [2]

Office has long been the predominant productivity suite for enterprise and home-based users but the release of a collaborative platform is a "game changer," according to Stephen Kearns of iMason Inc. [4]

Curious to see what some of the new features look like? Check out the gallery hosted by Microsoft [3]. Interested in finding out more about how Office 2010 can help you and your organization improve team productivity? Contact us directly at info@imason.com or (416) 597-3256.

[1] http://www.itbusiness.ca/it/client/en/home/News.asp?id=53868&PageMem=2

[2] http://www.itworldcanada.com/a/Daily-News/f83346b9-5007-43c5-b3f0-cd10ccd5b45d.html

[3] http://www.microsoft.com/presspass/presskits/2010office/imageGallery.aspx

[4] http://www.itbusiness.ca/it/client/en/Home/News.asp?id=53871

Celebrating 10 Years Building Web Technology That Works

Kerri McKenna — Mon, 13 Jul 2009 19:59:00 GMT

May 1st marks the start of 10 years in business for Toronto-based imason inc. Starting out in a humble office (i.e. basement) in Etobicoke, imason’s first paid project was for writing an article for Microsoft’s MIND magazine titled, “Make Your Legacy Apps Work on the Internet”. Since then we’ve grown to 40 people in a renovated space in the Entertainment District working on projects with many of Canada’s top companies. There have been numerous accomplishments worth noting over the years – awards, customer wins, technology firsts, and recognition from many sources. And all the while we’ve been teaching and helping our clients, and learning from them too.

Founders Jeff Dunmall and Scott Howlett sought to create ‘the best place to work’ in 1999, and have succeeded in attracting top talent who exemplify imason’s original mantra: experience, focus and passion. Although we’re a diverse team of skilled professionals who have many interests outside of imason, we’re bound by an enthusiasm for learning. We’re voracious consumers of information and we love solving the tough problems related to business IT strategy, programming, infrastructure and user experience.

We’ve got a lot to be proud of as a team, and you can bet that we’re heartily celebrating reaching the 10 year mark with Jeff and Scott. Every single person at imason helped us get here, and contributes to accomplishments that enable us to celebrate our core purpose: To revel in dramatically impacting our customers, our company, and each other. But we also wouldn’t be here without our valued customers, some of whom we’ve worked with for many years and others who we’re just getting to know on new projects. The relationships we’ve formed with you, and the challenges you’ve trusted us to solve, are two other important reasons we look forward to getting out of bed and coming to work every morning.

Cheers to our customers, our company, and each other!

Office 2010 Technical Preview Announced Today

Stephen Kearns — Mon, 13 Jul 2009 19:14:10 GMT

Hi All – I was at the Microsoft Technical Preview Press Event this morning to participate in a panel discussion with members of the press. Session was hosted by Jason Brommet and the good folks at High Road communications.

ITWorld Canada picked up the event and some of the partner discussion:

http://www.itworldcanada.com/a/Daily-News/f83346b9-5007-43c5-b3f0-cd10ccd5b45d.html

Here were a few quick hits on stuff that jumped out at me that I’m going to really enjoy in the new version.

Outlook
1. Convert voicemail to text (when hooked up to exchange)
2. You can mark a given conversation thread to ignore further updates
3. When you receive invites, body of message includes a calendar preview
4. Mouse-over emails/presence jelly-beans pops up a little contact-preview
5. New pane listing recent contacts you’ve interacted with (quick click to IM or email)
6. Quick-steps – kind of like a section on the ribbon where you can stick in some canned/customized macros – e.g. Reply with Invite – takes an email you’re on and creates invite with all the people on the email as invitees. Nice!
PowerPoint
- video as “first class citizen” – basic video editing and effects tools built right in
- more powerful image editing
- Record your PowerPoint presentation without having to go out to a 3rd party tool like Camtasia
- You can broadcast your presentation (pops up a provider list, including Windows Live for individual consumer audience)
Excel
1. Sparklines looks interesting – visualization/charting right inside a spreadsheet cell – may seem small, but when you’re pouring over volumes of data, the ability to visually highlight key points so person consuming the information can spot it quickly is really valuable
2. Slicers – ability to surface up your data filters on table or pivot tables as a little persistent dialog/window and see what filters have been applied, apply new ones, deselect.
Word
1. Backstage (across all apps) – surfaces lots of great doc properties/tools – e.g. integrates print dialog with print-preview
2. Ability to minimize the ribbon

You can see more screenshot info here: http://www.microsoft.com/presspass/presskits/2010office/imageGallery.aspx

Looking forward to seeing many of the other good stuff coming down the pipe including the web versions - “Office Anywhere” later this summer and the SharePoint side of Office 14 in the fall.

Graphic Design Guidelines for SharePoint 2007

Kerri McKenna — Wed, 08 Jul 2009 18:31:00 GMT

Problem: You've been asked to create a graphic design that can be applied to a SharePoint 2007 site - but you've never seen one before. Or, you've seen one but don't have access to the CSS to see how things are set up.

Here are some basic guidelines for creating a graphic design that can easily be applied to out-of-box Microsoft SharePoint 2007 sites. The guidelines apply to designs that will be implemented using SharePoint “themes”, which means that only CSS and graphics can be altered but the HTML on pages cannot. This is by no means an exhaustive list of SharePoint elements - but if you're trying to work on a mock-up of new graphic design, hopefully this will help clarify some of the terminology and limitations you might hear about when discussing the feasibility of implementing the design with a technical team.

Disclaimer: When I say that something "cannot" be done - I don't mean it's impossible, but rather that it's not easily done by modifying a theme file alone. And I won't claim to be a CSS guru either. I have no doubt that people have figured out ways to overcome some of the points below with more crafty CSS than I am capable of.

PAGE WIDTH

Pages have a fluid width and are designed to occupy 100% of the width and height of the browser window.
Pages are optimized for a 1024x768 browser resolution.

Tip
If designing with a “grid layout”, commonly used in print design, percentages will have to be used to define the column and gutter widths.

NAVIGATION

Tabs

Unselected tabs all look the same.
The selected tab can have a distinct look from unselected tabs.
You cannot single out a tab and change its style (e.g. adding a “New!” icon or unique colouring that draws attention to it).
The width of each tab defaults the width of the name of each tab, plus padding of XX pixels on either side of the text.
The widths of the tabs can be made consistent through styles, but longer tab names should be considered when defining the width.
The small arrow icon is a graphic, and shows up when a tab has menu items underneath it.

Left Navigation

In Sharepoint terminology, the left navigation bar is referred to as the “Quick Launch” bar. The links in this area are generated dynamically as users add lists and libraries to their Sharepoint site.

The width of the left navigation area is static and set in pixels. The width is the same on all pages.
Headings all share the same styles.
Links under headings all share the same styles.
The “View All Site Content” link can have its own style.
The Recycle Bin area can have its own style.

Breadcrumb

The breadcrumb uses plain text, hyperlinks, and the “>” character to let users know where they are in the site hierarchy.

Links can be styled.
Plain text can be styled.
The “>” character cannot be changed.

Global Links

Global links appear at the top of each page.

TITLES

Area / Site Title

The area/site title appears at the top of every page and is a hyperlink which can be styled.
The height of the white background seen in the screen shots can be modified as required.
The white background behind the area/site title, logo icon/graphic, and search bar can be styled.
The position of the title within the white background can be controlled with styles.

Logo / Icon

The default icon can be replaced with a JPEG, GIF, or PNG file.

The white background around the logo will increase or decrease in height depending on the size of the replacement graphic.
The icon can also be hidden and replaced with a background image that contains logos or graphics.

Page Title

The page title is plain text that can be styled.

Page Description

The page description is plain text that can be styled.

Consider a link colour for the page title, because on some pages the title may be a hyperlink.

WEB PARTS

“Web part” is a SharePoint term that in other platforms means “widget”. A “web part page” could be considered as a dashboard view of various lists consolidated on a single page. The condensed view of each list is referred to as a “web part”.

Web Part Title Bars

Web part title bars all use the same styles.
Sometimes the text is a hyperlink to the full view of a list; other times it’s just plain text.
The arrow icon on the right side of the title bar cannot be changed. When hovered over a drop-down menu appears.

LIST VIEWS

Toolbar

The toolbar options change depending on the type of list you are using, but the same styles are used everywhere.
The “View” option on the right side of the tool bar can have its own unique style.

SEARCH RESULTS

There may be more than one search scope available, in which case tabs are used to indicate the available options.
The title of the result is a hyperlink.
The URL for the result is a hyperlink.
The paging options are hyperlinks.

CONTENT PAGES

Article Page

The article title is a mandatory fields and shares the same style as all page titles.
The article date is an optional field that can be styled.
The article byline is an optional field that can be styled.
Content is entered through a rich text editor (RTE). Default styles for RTE text can be modified, and custom styles can be added.

Online forms - inline labelling

Aaron Bernardo — Wed, 24 Jun 2009 14:49:00 GMT

Any site or app that collects user information of any kind will more than likely require that a form be filled out. This is never the most fun part of the user experience but it is a necessary step in the process. If an online form has any obstacles there is a high probability that the user will either; contact someone and ask for assistance, bail on the process, or leave and never return. Therefore, anything that makes the entire process easier on the end-user can only have a positive outcome for everyone involved.

I’m interested in any tips that help the form completion process, these are never drastic additions that make the user's squeal with excitement, like a well developed RIA (Rich Internet Application) such as Flash or Silverlight, but… “It's the little details that are vital. Little things make big things happen.” – John Wooden

Here’s a nice tip I came across when creating forms…

http://www.zurb.com/article/271/making-forms-convert-through-awesome-inli

Inline labelling that remains even after the user has selected the input field, then disappears as the user types.

Usually the inline label disappears after it is selected which can cause problems if the label contains instructions or it is selected by default.

This effect is achieved through a combination of html, css and javascript.

It is being used with good results here …

www.me.com/

SharePoint FBA and the 403 Forbidden Error

Bob Brown — Sun, 14 Jun 2009 03:05:00 GMT

Scott alluded to the issue here (http://www.imason.com/blogs/scott_howlett/archive/2009/04/03/ie8-and-crm-3-0-unsupported-browser-or-is-it.aspx), but I’m going to take a deeper dive for all of the SharePoint users experiencing the following problem.

Set-up: SharePoint 2007 using Forms Based Authentication (FBA)

Problem: When trying to access the root site URL to log in, some users experience a blank screen or a 403 Forbidden error instead of the forms login page.

Reason...and it might shock you: Because of a conflict between the new Office Live Components and a by-design condition in WSS 3.0’s implementation of WebDAV, SharePoint gets confused, thinks that the web request is for WebDAV and returns a 403 error code.

Let’s look a little deeper...

If a user has installed the new Office Live Components (http://www.microsoft.com/downloads/details.aspx?FamilyID=91fe0002-eb00-434b-8726-27911326d2b2&DisplayLang=en) two new entries will be added to Internet Explorer’s user agent on each GET request: OfficeLiveConnector.1.3 and OfficeLivePatch.0.0

GET / HTTP/1.1

Accept: image/gif, image/jpeg, image/pjpeg, application/x-ms-application, application/vnd.ms-xpsdocument, application/xaml+xml, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, application/x-silverlight, application/x-silverlight-2-b2, */*

Accept-Language: en-ca

User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618; MS-RTC LM 8; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)

Accept-Encoding: gzip, deflate

Host: mysharepointsite.com

Connection: Keep-Alive

But the WebDAV implementation in SharePoint looks specifically for the substring “Office” in the user agent to make some assumptions about the request:

From: http://msdn.microsoft.com/en-us/library/cc250199(PROT.10).aspx

<4> Section 2.2.8: The WebDAV server in Windows Windows SharePoint Services 3.0 technology checks the product tokens in the User-Agent request-header field sent with a request for the presence of "Mozilla" and the absence of either "Office", "FrontPage", or "non-browser" to determine whether to send an HTTP 302 redirect message to a login page when using ASP.NET Forms Authentication for unauthenticated clients. Otherwise, the server assumes that the client user agent is not a browser and will return either an HTTP 401 "Unauthorized" error to prompt authentication when using Windows Integrated authentication, or an HTTP 403 "Forbidden" error when using ASP.NET Forms Authentication along with an X-MSDAVEXT_ERROR message [MS-WDV] section 2.2.3 with an Extended-error value of 917656. This specific error combination is used by the Windows client WebDAV Redirector as a signal to send a Cookies request-header along with the request. If the Windows client does not have a Forms Authentication cookie, the server will resend the HTTP 403 error with the X-MSDAVEXT_ERROR message with an Extended-error value of 917656, as a signal to the Windows client that it needs to prompt the user to authenticate.

So there you have it: the new Office Live Components add new entries to the IE user agent, but WebDAV confuses these entries with requests coming from a WebDAV compatible client (not a browser), causing an error to be returned.

I’ve come across two other people with the same issue, so you can check out the following links for more details:

I was really hoping that WSS 3.0 SP2 would have patched up this issue, but when I installed it recently, sadly no fix.

Workarounds: Unfortunately, I still haven’t come across a silver bullet solution for this but there are a couple things that you can do (I’m open to other ideas too if you want to suggest them in the comments):

1) Get your users to hit the FBA login page directly; for instance, instead of going to http://mysharepointsite, get them to go to http://mysharepointsite/_layouts/<FBALoginPage>.aspx?ReturnUrl= (notice the ReturnUrl query string parameter -> it’s mandatory)

2) Get your users to change their user agent by uninstalling the Office Live Components or using a tool like the User Agent Picker: http://www.enhanceie.com/ietoys/uapick.asp. This isn’t an ideal approach, but it does work

Hope this helps someone else!

MOSS: Indexing HTML from a Publishing HTML Field

Suleman Ibrahim — Tue, 09 Jun 2009 04:21:00 GMT

I was working on a custom search (Full Text Search) web part when at a point I came to know that managed property of a Publishing HTML field is returning just Text (simple) not actual HTML saved in that field. Whereas client requirement was to show formatted text with hyperlinks etc. in the web part. Initially I thought that there might be something wrong with either crawl or managed property for the field. However, I ran some tests and figured out that there is nothing wrong with crawl or managed property. So only other thing was, may be there was something wrong the way I setup/created Publishing HTML field.

For testing purpose, I created a new Publishing HTML field add some contents in that field, create crawl and managed properties then did a search against that new property and I was surprised new property was returning HTML instead of simple text… see screenshots below;

Added contents in both fields (having some bold, underline text)
Ran incremental crawl and see following results in the custom web part (I had already changed the web part query and XSLT to show both fields in the web part)

In this screenshot you can see Test Additional Text (New Field) is showing formatted text but Additional Text (Actual Field) is showing simple text.
Then I ran same query in SPQuery Tool and got following results that described the difference in both fields.

In this screenshot you can see Additional Text is returning just Text whereas Test Additional Text field is returning html.
Now this was confusing for me; apparently I wasn’t able to see any difference the way both fields, crawl and managed properties were being created. But then I got the difference and that was Face value of both field is different in the List view form.

May be some of you are thinking that AllItems.aspx looks ugly but in my case only Site Owners and Administrators can go to this page, all other users see results in the custom search web part on home page.
So yeah! I got the clue; actually problem was Crawler crawls face value/text not the hidden value/text that’s why I was facing that problem. But as I mentioned earlier there wasn’t any difference the way both fields were created.
I wrote a utility to compare all of properties of both fields --- Hurray :), I found something; that is RichText property of Additional Text is set to true, which make sense to show a nice and pretty view on AllItems.aspx page but as mentioned I wasn’t worried about AllItems.aspx.
I wrote following lines of code to set RichText property to False.

Microsoft.SharePoint.Publishing.Fields.HtmlField field = (Microsoft.SharePoint.Publishing.Fields.HtmlField)list.Fields["Additional Text"];
field.RichText = false;
field.Update();
Finally! my web part shows formatted text instead of simple text.

I hope this can save someone’s bunch of time who is facing same kind of problems with publishing fields :).

Retrieving source code from a specific TFS label

Peter Bojanczyk — Mon, 08 Jun 2009 14:07:53 GMT

Technorati Tags: TFS,Team Foundation Server,Visual Studio 2008,Labels

Grab a specific version

Choose Label and make sure both checkboxes are set (this ensures that your current Source code gets overwritten by the code you’ll pull from TFS), then click the “…” button (marked by arrow)

Choose your project and click Find. After that highlight your label and press Close. This will put the label text back into the previous screen

Your screen should now look something like this.

Click Get and you’re done!

Configuring Enterprise Library 4.0 for Exception handling and Logging

Suleman Ibrahim — Mon, 25 May 2009 21:18:00 GMT

A few weeks back, I was trying to configure Enterprise Library 4.0 for Exception handling and Logging exception into event log but was not able to find any help on internet so it took me long time to do all the configurations for handling and logging exceptions using Enterprise. Library 4.0.

Follow following steps to configure Ent. Lib 4.0 Exception Handling and Logging blocks.

Configurations:

Download and install Enterprise library 4.0, you can download it from here.
After installation go to Start -> All Programs -> Microsoft patterns & practices -> Enterprise Library 4.0 – May 2008 -> Enterprise Library configuration
A new wizard console will open then go to File -> Open Application -> (Navigate to your SharePoint web Application web.config) and click Open button.
After loading web.config, right click on the web.config path (see screen shot) –> New -> Exception Handling Application Block
A new block Exception Handling Application Block will be added to left pan of window.
Right click on Exception Handling Application Block –> New –> Exception Policy
A new sub block will be added click on Exception Policy and type some meaningful name for your project in right pan of window

Note: Exception Policy will change to new name you had written in this step in left pan as well. This step is the most important step, if multiple developers are working on the project you need to make sure everyone is using same name across the board otherwise you can run into issue while integration.
Right click on My Custom Exception Policy(left pan) –> New -> Exception Type
A new dialog window will open look for Exception and click OK button.
A new sub block Exception will be added to left pan.
Right click on Exception –> New –> Logging Handler
Another sub block Logging Handler will be added as well as a Logging Application Block. So left pan will looks like:
Click on Logging Handler and make following changes in Properties Pan.
Click on Formatter Type -> click … button (in value field)
A dialog window will open select TextExcetionFormatter then click OK
Select General in for LogCategory field.
Click on Formatted EvenLog TraceListener and change source to something more meaningful e.g. Application ABC Exception

Note: This step is not important but it would be good if team uses same Source name across the board.
Go to File -> Save Application to save all the configurations.

Web.Config Configurations:

Note: web.config changes are required if you are using Enterprise Library with SharePoint for .Net application you can skip these steps.

Copy following DLLs from “C:\Program Files\Microsoft Enterprise Library 4.0 - May 2008\Bin” and paste them into your web App Bin directory or deploy into GAC.

a. Microsoft.Practices.EnterpriseLibrary.Common.dll
b. Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.dll
c. Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.Logging.dll
d. Microsoft.Practices.EnterpriseLibrary.Logging.dll
e. Microsoft.Practices.ObjectBuilder2.dll
f. Microsoft.Practices.Unity.dll
Add following safe controls in web.config

a. <SafeControl Assembly="Microsoft.Practices.EnterpriseLibrary.Common, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Namespace="Microsoft.Practices.EnterpriseLibrary.Common" TypeName="*" Safe="True" />
b. <SafeControl Assembly="Microsoft.Practices.ObjectBuilder2, Version=2.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Namespace="Microsoft.Practices.ObjectBuilder2" TypeName="*" Safe="True" />
c. <SafeControl Assembly="Microsoft.Practices.Unity, Version=1.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Namespace="Microsoft.Practices.Unity" TypeName="*" Safe="True" />
d. <SafeControl Assembly="Microsoft.Practices.EnterpriseLibrary.ExceptionHandling, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Namespace="Microsoft.Practices.EnterpriseLibrary.ExceptionHandling" TypeName="*" Safe="True" />
e. <SafeControl Assembly="Microsoft.Practices.EnterpriseLibrary.Logging, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Namespace="Microsoft.Practices.EnterpriseLibrary.Logging" TypeName="*" Safe="True" />
f. <SafeControl Assembly="Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.Logging, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Namespace="Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.Logging" TypeName="*" Safe="True" />

Code:

Add following references and namespaces in your project

Microsoft.Practices.EnterpriseLibrary.ExceptionHandling
Microsoft.Practices.EnterpriseLibrary.ExceptionHandling.Logging

Note: Use Browse tab while adding reference and add above references from “<System Drive>:\Program Files\Microsoft Enterprise Library 4.0 - May 2008\Bin”

User following code snippet for exception handling:

try

{

//CODE

}

catch (Exception ex)

{

bool rethrow = ExceptionPolicy.HandleException(ex, "My Custom Exception Policy");

// Policy Name should be same as we configured in configuration step 7

if (rethrow)

throw; //Throw Exception if you need to show an error message to end user otherwise no need to throw exception.

}

finally

{

// Close/Dispose all objects

}

You are all done to get benefits of Enterprise Library 4.0 – Enjoy!!!!!

MOSS 2007 - Uploading and Indexing Large Files:

Suleman Ibrahim — Mon, 25 May 2009 18:13:00 GMT

A client has reported an issue that users were not able to upload files larger than 50MB file which we can resolve easily by updating Web Application settings; however, users were getting Timeout exceptions as well. Furthermore, business told us that business requirement is to be able to upload up to 100MB files.

There are few simple configurations setting those needs to do in order to upload large files in to SharePoint document libraries….

SharePoint Configurations:

Increase the maximum upload size
The maximum size for uploading files is set to 50 MB for a Web Application. If you need to be able to upload larger files, you can change this setting to any value up to 2 GB (2047 MB). However, as this is web application level setting so even if you have multiple site collections, this change will replicate to all site collections.

1- Go to Start –> All Programs -> Administrative Tools –> SharePoint 3.0 Central Administration -> Applications Management.
2- Go to Web application general settings under SharePoint Web Application Management section.
3- Look for Maximum Upload Size and update that field as required up to 2 GB (2047MB)
4- Click OK to save Settings.
Increase the default chunk size for large files
The chunk size is not related to the maximum upload file size. The chunk size simply specifies the amount of data that can be read from a file at one time. By default, the large-file-chunk-size property is set to 5 MB. If you notice performance or scale problems on the client or server, then you may need to tune this setting to get the performance you are targeting.
Note: If you raise the chunk size too high, the files might use up too much front-end memory and you may need to lower this setting.

The large–file–chunk–size property must be set from the command line. This property is configured for a server or server farm, and cannot be configured for an individual virtual server. To set this property, use the following syntax:

1- Open Command Prompt
2- Navigate to 12 Hive Bin Folder (usually C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\Bin)
3- Run Command: Stsadm.exe –o setproperty –pn large–file–chunk–size –pv <Size in Bytes>
4- After making a change to this property, you must restart IIS. You can restart IIS by typing iisreset on the command line.

Tune the Microsoft Internet Information Services (IIS) connection timeout setting:

By default, timeout for is set to 120 seconds (2 minutes). Depending on your maximum file size and how long it takes for the file to be uploaded, you may not need to change this setting. If, however, IIS is timing out when you upload large files, you can change this property to ensure that larger files can be uploaded successfully.

1- Go to Start –> All Programs -> Administrative Tools –> Internet Information Services (IIS) Manager.
2- Right-click the virtual server you want to configure, and then click Properties.
3- Click the Web Site tab.
4- In the Connections section, in the Connection timeout box, type the number of seconds you want IIS to wait before timing out.
5- Click OK to save settings.

Those who are working in SPS 2003 and want to make changes for web parts, please have a look to this article and find Configuring Large File Support.

Web.Config configurations:

Update maxRequestLength and executionTimeout attributes for the httpRuntime node:
excecutionTimeout = <Time in Seconds>
maxRequestLength = <File size in KB> (by default for ASP applications, its set to 4096 (4MB), but because SharePoint supports 50MB so its set to 51200)
e.g.
<httpRuntime executionTimeout="999999" maxRequestLength="102400" />

Indexing Large files:

If large files needs to be crawl then we need to make following changes in the Registry and central administration.

Registry Configurations:
By default, Search Services can crawl and filter a file with a size of up to 16 megabytes (MB). It will always crawl the first 16MB of a file. After this limit is reached, SharePoint Portal Server enters a warning in the gatherer log “The file reached the maximum download limit. Check that the full text of the document can be meaningfully crawled.”
To increase the limit of 16 MB, you must add in the registry new entry MaxDownloadSize. To do this, follow these steps:

1- Start Registry Editor (Start –> Run –> Regedit).
2- Locate the following key in the registry: HKEY_LOCAL_MACHINE -> SOFTWARE –> Microsoft –> Office Server -> 12.0 –> Search –> Global –> Gathering Manager
3- If already MaxDownloadSize exits then move to step 5.
4- Right click –> New –> DWORD Value. Name it MaxDownloadSize.
5- Double-click on MaxDownloadSize, change the value to Decimal, and type the maximum size (in MB) for files that the gatherer downloads.
6- Restart the server (this is an important step to do, otherwise changes will not effect).
7- Start Full Crawl.

NOTE: Increasing the file size may cause a timeout exception because the crawler can timeout if the file takes too long to crawl/index (because of its size). To increase timeout value, follow these steps:

Central Administration Configurations:
1- Go to Start –> All Programs -> Administrative Tools –> SharePoint 3.0 Central Administration -> Applications Management.
2- In the Search section, click Manage search service
3- On the Manage Search Service page, in the Farm-Level Search Settings section, click Farm-level search settings.
4- In the Timeout Settings section change Connection and Request acknowledgement time. (here you need to do your math that how big files are suppose to be index and approximately how long it will take to crawl – for 100MB I haven’t change it so it’s still default 60 seconds for both).

You are all done and ready to test your scenarios :).

Enjoy!!!

The Problem: Why Employees Can’t Find What They Are Looking For

Jeff Dunmall — Wed, 20 May 2009 14:34:00 GMT

It is easy to create content. Fire up a Word document and post it, or create a sub-site on your Intranet and add pages. Even creating a new web site isn’t hard anymore – most business users can go through a process to launch one quickly. The benefit: quickly converting corporate knowledge into something that can be consumed by others. The challenge: it is up to the end user to ‘know’ where to find it, and that often doesn’t happen.

There is good news. Today’s search engines make it easy to hook up new sources of content. Check out the solution you have today for internal search and hunt around for a dropdown box of options. There are probably dozens. Each time a new web site comes online internally, the search engine needs to be pointed at it. It probably includes: HR content, news, product information, service information, people directory, manuals & procedures, links to tools, potentially file shares, email, collaboration environments like SharePoint, plus perhaps a Documentum or Interwoven, and likely multiplied by geography, business units, secured content, or all of the above.

The user is faced with two options: 1) choose a source, or 2) search everything. Lets look at each in more detail.

1) Choose a Source – this option requires an employee to know what a three or four word description means, and what kind of content is likely to be included. It also assumes that it is possible to name the content, and that generally a single type is available. As that is often not possible (consider how many different types of content you have on your ‘Intranet’), the sources get redefined to be more granular. The result: many more options to choose from. Realistically end users cannot be expected to hunt through this list to find the right one, which leads to most choosing the second option.

2) Search Everything – This is often the default: search across all content under index and provide a result set. That can include hundreds of thousands of items, and as you get all of your content under index it can easily span millions of entries. This option assumes two things. First, that your search engine can bring back the top 10 relevant results every time based on a few words of input, and second that each result should be displayed in the same format (typically, title, link, preview, plus potentially a date). Do those assumptions seem reasonable?

So, what to do? More on that soon.

VSTS – Setting Permissions on Work Items?

Stephen Kearns — Tue, 19 May 2009 19:11:00 GMT

Is it possible? I was thinking the answer was no – but I poked around and ran across this forum entry which outlines how you do it – basically, the short version is TFS allows you to apply permission (or deny permissions) to a specific Node in the Area Path Tree.

http://social.msdn.microsoft.com/Forums/de-DE/tfsworkitemtracking/thread/280095b2-974e-469e-b879-9ec9f1bacfb0

Steve Ballmer on assessing job candidates

Scott Howlett — Tue, 19 May 2009 16:14:07 GMT

This is from a great interview of Steve Ballmer done by the NY Times (bolding is done by me):

Q. How do you assess job candidates?

A. If they come from inside the business, the best predictor of future success is past success. It’s not 100 percent, but it’s a reasonable predictor. For an external candidate, what I’ve found is that reference checks are super-important. I didn’t used to believe so much in reference checks. You can always get somebody to say something nice about you. But the truth is, if you ask enough questions and you ask around, you can really get a profile of who’s accomplished various things and who hasn’t.

And I try to figure out sort of a combination of I.Q. and passion. I just ask somebody to tell me what they’ve done that they are really proud of and tell me about it. And if it’s something you are proud of, you should be able to answer any question I can come up with, at least at a level that would satisfy my interest. I ought to be able to see your passion. It might be quiet passion; it might be bubbly passion. But I should be able to sense that you are one of those people who just sort of throws themselves into things.

I get asked this question all the time, particularly in forums like in Microsoft’s Ignite Your Career webcast series and I love Ballmer’s response. My answer is very similar to Ballmer’s, except I usually combine “Specialism” with “Passion”. That is, talk to me about where you’ve specialized and why. And in doing so, I should be able to get a good sense of your passion for your specialty. We’ve long recruited for passion at imason, and indeed one of our Core Values is “Be Passionate”. At imason, we’ve had interviews where we spoke at length about such things as the Sitar (we hired this person) or even medieval German Literature (the interviewee had a Masters in the field, but we didn’t hire this person). We always try to get people to talk about something they’re truly passionate about, and it doesn’t need to be work-related and in fact, it usually isn’t.

You see, the thing is that if you’re a passionate person you’re highly likely to be passionate about everything you do including your work.

It’s a great interview with Ballmer – definitely check it out! And Be Passionate!

Enterprise Search: Does the Name Help or Hurt?

Jeff Dunmall — Fri, 15 May 2009 16:08:21 GMT

If you’re looking for an internal search solution you’ve probably heard the term Enterprise Search kicking around. It generally applies putting in an internal search solution that can broadly index content (file shares, email, SharePoint, etc) and provide a search interface for your employees.

It assumes a few things. First, that the solution applies to all employees. After all, don’t all employees need the ability to search through their knowledge work? Isn’t search just like email? The key flawed assumption: all employees share the same underlying search problem.

I certainly agree from a strategy perspective. An Enterprise Search Strategy can help identify the holistic approach to rolling out and implementing a search solution. That may be where the usefulness of the term ‘Enterprise’ ends.

I think it is more constructive to think of search in the way you currently think of your database platform. Don’t databases require real business applications to be useful to their end users? Would you attempt to justify the purchase of a database without the needs of a specific group of users in mind? Search solutions should be approached the same way – they have a specific set of users in mind, and the solution should be tuned to provide the best search experience possible.

Here is a clear symptom of that problematic assumption: are you having a hard time building a Enterprise Search business case? If the search solution is inexpensive, like a Google Search Appliance or Microsoft Search Server, then a business case isn’t as important and you probably are using the ‘we just need it’ argument to justify the expense. That’s valid. However, if you are spending significantly more on a platform like FAST or Autonomy, you need a solid business case. One of the best ways to accomplish that is to know specifically who the solution is for, and precisely what benefit you’ll drive by deploying it. That concept contradicts the ‘Enterprise’ assumption we often start with.

My suggestion: look for a group of users that share a common knowledge discovery challenge, and work on building a search solution and a business case around them. Look for a group that is large enough to support a business case, but not so large that their job functions diverge and therefore their search needs are too broad. Also, establish an overall Enterprise Search Strategy so this and future search solutions can fit into a cohesive plan.

Event ID: 6875 in Event Viewer after deleting an SPFolder in a SharePoint document library

Noorez Khamis — Thu, 14 May 2009 22:20:00 GMT

ehucsd969j

I am deleting a folder from a document library using the code:

using (SPSite tempDocLibSite = new SPSite(tempDocLibWebRootUrl))
{
    using (SPWeb tempDocLibWeb = tempDocLibSite.OpenWeb())
    {
        tempDocLibWeb.AllowUnsafeUpdates = true;

        SPFolder tempAttachmentSubFolder = tempDocLibWeb.GetFolder(string.Format("{0}/{1}/{2}", 
            tempDocLibName, base.ListId.ToString(), hdnTempAttachmentID.Value));

        tempAttachmentSubFolder.Delete();

        tempDocLibWeb.Update();

        tempDocLibWeb.AllowUnsafeUpdates = false;
    }
}

When it get’s to the highlighted line, I get the following error in the event viewer with Event ID: 6875:

Error loading and running event receiver Microsoft.SharePoint.Publishing.Internal.PublishingWebEventReceiver in Microsoft.SharePoint.Publishing, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c. Additional information is below.

: There is no file with URL 'TempAttachmentsUpload/e4486bd9-1719-4a9d-a8a1-de25924cd49a/0136561e-0d20-4b0d-b7e7-ab1e1ada6218' in this Web.

The funny thing is that the folder does exist and does get deleted.

There are lots of resources on the web about this event id for SharePoint but it looks like it’s an event that is very common for different reasons.

I am using MOSS SP1 with the December Cumulative Updates.

It looks like there is a built-in event receiver that is executing when the folder gets deleted which is causing this error. I don’t think it should cause this error and it looks like it’s a bug, hopefully someone can confirm it or correct my code. I’m not sure if it is fixed in SP2 or the April Cumulative Updates though.

[1] Technorati Profile

Make Your Legacy Apps Work on the Internet

Scott Howlett — Mon, 27 Apr 2009 21:30:23 GMT

So, we’re just retiring our first server. We used to name our servers after our clients, and our first client was MIND – Microsoft Internet Developer magazine, a pre-cursor to MSDN magazine. It was with nostalgia, that I went looking for that first bit of technical prose… But alas, the Microsoft archives only go back to 2000… Duh??!!

Not to be discouraged, I turned to the amazing web archive and they had the copy. Amazing!

So, in honour of the now defunct MIND server, I reprint the original “Make Your Legacy Apps Work on the Internet” authored more-or-less 10 years ago, and still right on message. You have to love the old days when “VB6 COM Objects” were cutting edge, and XML was the be-all-and-end-all-that-it-should-be-the-whole-cover. Beautiful.

Check it on the web archive.

----

This article assumes you're familiar with XML, Microsoft Message Queue, and Visual Basic
Download the code (22KB)

Make Your Legacy Apps Work on the Internet
Scott Howlett and Jeff Dunmall

The Internet is increasingly becoming an important path for business-to-business data services. You can use BizTalk and COM to keep your existing systems useful for longer.

In the coming years, most companies will integrate their Internet presence with their mission-critical line-of-business systems. Creating these applications will be the most difficult challenge Internet developers have yet to face. It means enabling interoperability between legacy systems, possibly from different companies, and doing it with the availability and scalability that Internet applications demand. By getting into the right design mindset and making the right technology choices, notably Extensible Markup Language (XML) and Microsoft® Message Queue Services (MSMQ), Internet applications can provide the return-on-investment that has been promised in the past, but rarely delivered.
First, let’s explore what legacy systems are and why they’re important. We’ll walk you through interoperable Internet application development and the issues involved in building them. Then we’ll build an Internet-based Address Change facility as a traditional Internet application that integrates with existing systems. We’ll also show you how the design is extensible, enabling business-to-business communication using the BizTalk framework. Surprisingly, the framework can be written with only about 100 lines of ASP and Visual Basic® code.
Microsoft will expand the technology available for integration with legacy systems when it releases the Microsoft Enterprise Interop Server, codenamed "Babylon" (see Figure 1). Babylon delivers application integration with COM Transaction Integrator (COMTI) and an MSMQ-to-MQ

Figure 1: Microsoft Enterprise Interop Server Architecture

Series bridge, data integration with OLE DB providers and ODBC drivers, and network/platform integration via an SNA gateway or direct TCP/IP access. We’ll only address the challenges of getting data to the Interop server. To find out more about Microsoft’s interoperability strategy, go to http://web.archive.org/web/20000617215818/http://www.microsoft.com/isapi/gomscom.asp?target=/interoperability/.

Background

If you were at Microsoft TechEd this past May, you probably heard Paul Maritz’s discussion about the upcoming third generation of Internet applications. In his analysis, the first generation involved content publishing based on the HTML standard. The second generation was based on dynamic content using Windows® DNA technology, and the third generation will feature integrated Internet systems that enable more effective line-of-business applications.
Building third-generation Internet systems will demand integration with legacy systems from one or more companies. Companies will be aggressively developing these applications because they know that customers and suppliers will demand integrated systems based on open Internet standards. Increased efficiency and streamlined business processes should have pleasant effects on the bottom line, in part through the competitive advantage gained by providing better and faster service. If a company doesn’t or can’t provide integrated services, business opportunities will likely be lost.
If you read Don Box’s article, "Windows 2000 Brings Significant Refinements to the COM(+) Programming Model" (Microsoft Systems Journal, May 1999), you were probably surprised at the notion that COM components developed before Windows 2000 are often referred to as "legacy" components. You’ve never written any COBOL, so how could you have any legacy code?
The terminology shouldn’t surprise anyone. It simply means that you have written software that is part of a production system. Having legacy code is a good thing because the alternative is that none of your code is working in production! So, simply put, a legacy system is any system that exists in production. It’s made of legacy code and the manual processes, such as call centers, that support the system.
Legacy systems often have hundreds of development-years invested in them. This is why it’s so important to evaluate legacy systems and build Internet systems on top of their foundations. With the right design, you will be able to overcome the pitfalls associated with integration and interoperability.
Building line-of-business systems in Internet time (less than six months) demands that existing systems be exploited. Usually, legacy systems have some manual processing where the combined expertise and experience of people is critical. You simply can’t reproduce that effort in your timeframe, and you probably don’t want to anyway—unless you have a penchant for pain or like to rewrite COBOL source code.
Of course, legacy systems aren’t all good news for Internet applications. When you bring them into the fold, you’re adding the most dangerous software villain: the unknown quantity of legacy systems. But on the bright side, you’ve got an excellent starting point because you already have a working system.
Legacy systems are frequently not documented, leaving everyone afraid that doing anything to them might break them. You can’t change any of their code and no one can or will tell you how they work. So there have to be legacy experts on the team, at least for the design period. They’ll be responsible for investigating, researching, and documenting the interfaces, business rules, and other facets of the legacy systems. Finding legacy experts is usually the first problem in building these systems.
On the technology side—where we’ll focus the rest of this article—there are a number of hurdles to overcome. Integration isn’t always easy. In fact, it almost always involves some pain. But with the introduction of technologies such as XML and application services such as MSMQ, building interoperable systems has never been easier. Microsoft Windows 2000 offers more technology advancements that will make interoperability even easier, notably native support for queued components. Babylon, due in beta in the last quarter of 1999, will provide even more opportunity for legacy integration.

Availability, Performance, and Scalability

Generally, integration problems fall into three familiar categories: availability, performance/scalability, and exception handling.
The availability of your application is related directly to the availability of the legacy systems you connect with. Your Internet application likely needs to run 24/7, while your legacy systems may not have the same constraints. They may be scheduled for nightly maintenance or weekend shutdowns. What will your Internet application do when legacy systems are unavailable?
From a performance and scalability perspective, the legacy system probably can’t handle the volume of traffic that the Internet site can, at least not in the same form. Including legacy resources in your application increases transaction time, thereby decreasing scalability. Sometimes even connecting to the legacy systems can be a costly operation. If the only interface to the legacy system is through screen scraping (don’t laugh, it happens all the time), then you definitely have a back-end resource bottleneck to overcome in your design.
If you have any combination of these issues, you must find a way to decouple your legacy systems from your Internet application, making it transparent to your customers. The most effective way to accomplish this is with MSMQ. On the Internet side, you can concurrently send thousands of requests and get excellent performance from MSMQ. On the legacy side, you can process those messages at a pace that the legacy system can handle. If there are only five database connections available through SNA, for example, you can process five messages at a time. This effectively allows you to throttle the load on the legacy system, while at the same time addressing your availability concerns and allowing your Internet application to scale appropriately.
A third grade French teacher once said, "Pour toutes les règles, il y a des exceptions." (Every rule has an exception.) We don’t remember much French, but after years in the software business, we’ve refined these words of wisdom: "Every business rule that has existed for more than five years has at least one exception." And herein lies the third category of integration problem: if every rule can be broken, how do you write a middle tier that enforces the business rules? As a further complicating factor, many of these rules are entrenched in the legacy systems and manual processes (and people) that surround them. So to build a completely automated system, it’s necessary to discover every business rule that’s grown up around the process, and to uncover all the exceptions to the rules.
Realistically, it takes time to flush all of these rules out and code them into your application, and they typically aren’t documented very well. In many cases, this is not possible in a six-month timeframe, so you’ll need to incorporate the manual processes to handle the remaining exceptions, at least in the first version. The golden rule is to make most things automatic and everything else possible, even if it means invoking a manual process.
For example, if you move to another town, it’s likely that your auto insurance policy needs to be updated to reflect your new neighborhood. So when building an Internet address change system for an insurance company, you have two design choices. You can either reprogram the system to automate the adjustment of the policy, or you can integrate this change within the existing process.
If you’ve ever worked with an insurance company, option one should send shivers down your spine, especially if you have to deliver your application in six months. Changing a policy involves numerous rules that need to be enforced, many of them for legal reasons. Besides, you’re supposed to be building an address change system, not an automatic policy update system. So the best alternative is to integrate the existing policy update process into the new system. This will save you time (the scarcest resource) in the development cycle because you won’t have to discover every rule and exception up front.
By incorporating the existing manual processes, you can rely on the expertise of the people who know the system best. Integration makes your life easier by saving you time and headaches. In version 2, after you get the app up and running, you can seek to uncover and incorporate more business rules and thereby automate the system more completely. Don’t underestimate how complex this will be. Even though a division might look like a call center to you, it’s also the brain center for many businesses, and it can’t be easily replaced.

How to Bring Them Together

So how do you overcome the availability, performance/scalability, and exception handling pitfalls? How many new design patterns and technologies do you need to learn? There’s some good news. All three areas can be addressed by one technology choice: a combination of MSMQ and XML. But technology alone won’t save the day. To do it right, you’ll need to change the way you think about application development.
Generally speaking, people are on-demand thinkers. When we’re hungry, we eat; when we’re thirsty, we drink. This type of thinking translates into a design pattern that will leave you dead in the water when it comes to building integrated Internet systems. You have to break this design pattern by thinking asynchronously first, and designing synchronous transactions only as a last resort. If you must use a synchronous transaction, hold on to your hat—in some cases it simply may not be possible with hundreds of users. Benchmark synchronous transactions early to avoid late-breaking scalability problems.
You also have to get over the common mindset that goes something like this: "If I (re)build everything in the system, all the code will be mine and everything will work." While this may be true in some cases, it is not the proper approach, especially if you want to finish on time. The key here is to make use of the enormous amount of effort that has been invested in existing systems and processes. If you integrate with an existing process, it’s likely that the development time spent handling exceptions can be reduced by half because the existing process already has built-in exception handling mechanisms.
Finally, you must embrace the versioned approach to software development. If version 1.0 of your application is also meant to be the last version, it is almost certain to fail. A one-version approach will compromise proper system design, the first casualty of increased scope inside the same development schedule. Furthermore, you’ll lose the ability to make course corrections, both in overall architecture and specific features.

MSMQ and XML

As we mentioned previously, the three main problems with integrated systems (availability, performance/scalability, and exception handling) can be addressed with a single technology choice—MSMQ and XML.
Many systems already communicate by passing a simple string. In order for both systems to understand the format and location of the data in the string, it’s probably marked it up with some kind of token system or based on fixed-length fields. XML is the formalization of this concept. It also includes the tools and techniques to make development easier. By using valid XML—XML that conforms to a document type definition (DTD) or schema—industries can standardize on a data format, giving applications the ability to exchange data with a much larger audience. XML provides a great way to model data and represent that data in a simple and powerful format.
What XML does for data, MSMQ does for transport. MSMQ gives systems a reliable and disconnected path for the transfer of XML between different systems in an organization. It guarantees that the XML is delivered once (and only once) when network conditions permit. If you’re integrating with systems that are widely distributed geographically, not always available, slow, or nonscalable, MSMQ allows you to isolate these systems from your customers. Your customers need not be concerned with your legacy systems because customers never interact directly with them.

Sample Application

About a year ago, one of our cars was stolen from the airport. The settlement for my car showed up in the requisite 60 days, but a check for the car’s contents never arrived. After six months or so, we gave the insurance company a call. After all, the Smiths CD that was in the car had to be replaced. The agent said that the second check was indeed sent over five months ago. It turns out that while the auto policy had the new address, the home policy (which insured the contents of the car) still had the old address. The same insurance company held both policies, so the cause of this mishap was surely the lack of integration across their line-of-business systems.
Let’s examine an Internet-based address change facility that solves the problem we just outlined. The insurance company has many internal processes, but we’ll focus on these three parts of the call center:

Auto policy address change: when a customer calls, the change is entered into a database via a terminal session.
Auto policy change: when a customer calls, an email is sent to an insurance agent, who then reviews the policy and makes the appropriate changes.
Home policy address change: for a simple address change, a fax is sent to the auto policy division that handles the address change.

In our case, it’s likely the fax was never sent and thus my home policy address was never updated.
Let’s take a quick look at the typical Internet solution and highlight some of the pitfalls. We’ll then show you a good second-generation application and its extension to a third-generation app using the approach advocated by BizTalk.org, which was announced at TechEd in May 1999.

Typical Approach

The typical approach toward building an Internet-based front end over a legacy system would be to build an HTML form that posts to an ASP page. The ASP code would connect to each of the data sources and execute some SQL to make the address changes. Finally, you’d write some HTML back to the client, indicating the success or failure of the operation. The HTML form would look something like Figure 2.

Figure 2: A Traditional HTML Form

This solution does not address the availability, performance, scalability, or exception handling challenges. First, availability is not optimal because if any of the data sources are unavailable at submit time, the user will receive an error message. Furthermore, if the underlying data sources run on a mainframe (as most legacy systems do), there may also be transient problems with connectivity through the SNA gateway. And, of course, there are the usual network problems, which may be intensified if the database is located across the WAN.
There is a big problem with performance: the user is waiting for all database transactions to complete. This can be disastrous, especially at busy periods (end-of-month, holidays, and so on) when legacy systems typically run at or near full capacity. Having this many database connections on a single page also presents scalability problems, especially if connections are limited (which they frequently are in legacy systems) and if there are contention issues. There may even be a show stopper if the transactions are lengthy and are locking resources.
This system does not exploit existing systems, and there is no exception handling mechanism. There are also less obvious problems with this solution. What if complex business rules need to be enforced? Even worse, what if they had traditionally been enforced by a manual process? What if connectivity to the back-end database is simply not possible (existing instead in a flat file on the mainframe) or a subsystem is down for regular maintenance? What if you want to share this information with an affiliate company? This may sound like worst-case planning, but these are common concerns at large corporations.

A Better Solution

Now, let’s take a look at how this system could be built to address the shortcomings of the traditional approach. As you read, keep three "VIA" (Version, Integrate, Asynchronous) rules in mind:

Plan for another version (you don’t have to do everything in version 1.0).
Integrate existing systems and processes (don’t start from scratch).
Think asynchronous first (isolate existing systems).

From the user’s perspective, the interface is still an HTML form with the same appearance as the one shown in Figure 2. Instead of posting the HTML form, you’re going to represent the data in XML and post the XML to the server. An excellent source of information about XML was Dino Esposito’s June 1999 Cutting Edge column.
The project starts with a DTD, which defines the structure of the XML. The source code is shown in Figure 3. DTDs have their place, but they don’t describe data in a way that is useful to data architects. A new standard, schemas, is emerging. Schemas use XML itself to describe the XML document structure. We chose to use XML schemas for this reason. Schemas are also a key part of the BizTalk initiative, which was started to facilitate business-to-business communications using XML.
The DTD and schema serve the same goal: they allow the XML parser to validate an XML document against a reference to make sure it conforms to the published standard. An XML document that conforms to a DTD or schema is said to be valid. Schemas are currently only supported by Microsoft Internet Explorer 5.0. A sample Customer schema is shown in Figure 4.
Before we get into the nitty-gritty implementation details, let’s go over the entire solution first. We’re taking a straightforward, six-step approach (see Figure 5):

Generate XML on the client machine.
Submit the XML to the server.
Package the XML into an MSMQ message and send it to the Distributor.
Send XML confirmation back to client.
Based on data stored in the registry, the Distributor will send one or more additional messages (one for each registered system).
These messages will in turn be received by Handlers, which will initiate a manual process or execute an address change in a particular system.

Now that you have the schema for the XML document, let’s take a look at how to generate the XML document on the client (see Figure 6). Most of the work is done in AddXMLNode. Notice that the schema definition is added to each node. This ensures that every node in the document is bound to the XML schema. By setting the DOMDocument.async property to false and then checking the DOMDocument.parseError code, you can determine whether the generated XML is valid. The resulting XML document is shown in Figure 7.

Figure 7: Customer XML Document

At this point, you’ve defined your XML schema, created an XML document on the client, and validated it against the schema. Now it’s time to submit it to the server. To post the XML to the page, use the XMLHTTPRequest object included in the msxml.dll that ships with Internet Explorer 5.0. Using this object offers the best performance (by providing the leanest possible HTTP POST forms without compression) and the most straightforward code. The client-side source code to post the XML is shown in the SendXML method in Figure 6. Prior to Internet Explorer 5.0, the preferred method for HTTP posting would have been through the WinInet API. We’ve seen the code that does this, and it’s not pretty.
The code to receive the XML on the server is shown in Figure 8. Note the call to set the async property of the XMLDOM object to False. If you omit this line, the resolution of the schema and the eventual parsing of the document will be performed asynchronously, which is not what you want in this case. The source code that receives XML on the server is shown in the ProcessRequest method in Figure 8. The SendMessage function sends an MSMQ message to the Distributor here as well.
The address change message is sent from the ASP page to the source queue shown in Figure 9. When it arrives, MSMQ notifies a Listener that calls the Distributor, a Microsoft Transaction Services (MTS) component. The Distributor pulls the message off the queue and sends the XML body to the destination queues as specified in the registry. Should something go wrong while sending the messages, the transaction rolls back, leaving the message on the source queue and nothing in the destination queues. More detailed information about sending and receiving MSMQ messages is available in Ted Pattison’s article, "Using Visual Basic to Integrate MSMQ into Your Distributed Applications" (Microsoft Systems Journal, May 1999).

Figure 9: Distributor Architecture

To take a message off a queue in an MTS transaction, the application removing the message must be running locally on the same machine as the queue and it must be running in MTS.

The Listener

MSMQ notifies the Listener application (see Figure 10) when a message arrives by declaring an MSMQEvent object called WithEvents:

 Dim WithEvents msmqMsgEvent As MSMQEvent
 Dim msmqQue As MSMQQueue

To set up notification in the Visual Basic-based listener, you’d then use the following code:

 Set msmqInfo = CreateObject("MSMQ.MSMQQueueInfo")
 Set msmqMsgEvent = CreateObject("MSMQ.MSMQEvent")
         
 msmqInfo.FormatName = txtSourceQueue
 Set msmqQue = msmqInfo.Open(MQ_RECEIVE_ACCESS, MQ_DENY_NONE)
 msmqQue.EnableNotification Event:=msmqMsgEvent

Instead of using a path name here, you should use a format name to increase performance. Using a path name requires a query to the Message Queue Information Store (MQIS). That RPC call adds significant overhead to the open queue request. Using a format name, on the other hand, requires only a single RPC call to the MQIS (if it is not a direct format name). After the first call, MSMQ caches the connection information, which removes the site controller from the picture and increases performance. This will be particularly relevant in the MTS Distributor component.

Figure 10: The Listener Application

When an MSMQEvent_Arrived event is fired, the Visual Basic-based component calls the Distributor in MTS with the format name of the source queue. It does not pass a reference to the source queue directly; calling ReceiveCurrent on the reference would not include the source message in the transaction because the queue was not opened in the context of the transaction. If the transaction aborted, the message would neither appear in the destination queue nor remain in the source queue.

The Distributor

Based on data stored in a local database, the Distributor will send one or more additional messages based on the contents of the registry (see Figure 11). The source code for the Distributor is shown in Figure 12. The messages sent by the Distributor will in turn be received by Handlers, which will initiate and execute an address change for a particular system. The architecture of a Handler is very similar to the Distributor, so we’re leaving out the details.

Figure 11: Registry Values

In a more robust system, the Distributor might determine the destination based on a database lookup and the type of XML schema used in the body of the message. This more flexible architecture could be used to process other message types as well.
Figure 9 gives the impression that the Distributor and destination queues are all running side-by-side on the same machine. While this is possible, it is equally likely that the Handlers would be running in separate offices, maybe even in different countries. This transparency gives your application the ability to communicate over slow links or WAN connections, knowing that your message will be processed when network conditions permit.

Benefits of this Solution

Before going into the benefits of this architecture, take a look at the amount of code written for this sample. Granted, the sample app is straightforward, but the table in Figure 13 shows just how little source code is required to build the framework around an integrated Internet site.
As you can see in Figure 13, there is not much source code involved in our sample application. Remember VIA? This solution fulfills all three parts:

Versioned: the first version of this system was simple and could be delivered in a realistic timeframe. However, the design and technologies chosen allow for easy extensibility in the future (see the BizTalk section that follows).
Integrated: we did not start this project from scratch. For example, the existing fax-based submission process was incorporated rather than attempting to redo it from scratch. By doing this, we did not have to incorporate the business rules surrounding modifications to insurance policies.
Asynchronous: the system has very few synchronous processes and the legacy systems are isolated from the customer. This optimizes the availability of the system. Furthermore, the performance and scalability of the system will be exceptional because there are no synchronous database connections.

Let’s take a look at a possible extension of this simple application. Suppose an insurance company was going to provide an address change service as part of their policies. This service would update the customer’s profile at their bank at the same time. How would the insurance company be able to provide this service? This is where the BizTalk framework fits in. BizTalk is dedicated to establishing, publishing, and maintaining industry-specific XML schemas to facilitate the exchange of information between businesses. Instead of using a local XML schema, this application might instead use a schema located directly on the BizTalk site (http://web.archive.org/web/20000617215818/http://www.biztalk.org/).
The next step would be to register another data service with the Distributor. This data service would simply post valid XML—according to the BizTalk customer schema—to the bank, which would then process the request. In effect, the bank would not be able to tell if the XML request came directly from a customer or via his insurance company. Version 3 of MSMQ is slated to support native HTTP message delivery that extends the guaranteed delivery concept across the Internet (version 2 is scheduled for release as part of Windows 2000).
Figure 14 extends the conceptual diagram of the system, making this a business-to-business solution. As you can see, extending a second-generation application to include business-to-business operations is quite natural—especially if the BizTalk framework is used to standardize on the use of a single XML schema.

Conclusion

To build third-generation Internet applications, you’ll need to build an interoperable system. Look at legacy systems and processes as opportunities to be exploited because the technology exists to address their shortcomings. By making the right technology choices now, communication with the systems of other companies will be a natural extension in future versions.
It’s clear that XML is emerging as the standard data format for communication between systems. On the transport side, MSMQ is an easy-to-use, asynchronous mechanism that can increase both performance and scalability. It will also isolate legacy systems and increase the availability of the system.
To build this new brand of applications, you’ll need to focus on the VIA design techniques we discussed in this article. Using these technologies and the VIA design concepts, you should be able to find the way to your third-generation Internet application. Good luck!

From the September 1999 issue of Microsoft Internet Developer. Get it at your local newsstand, or better yet, subscribe.

http://web.archive.org/web/20000617215818/http://www.microsoft.com/isapi/gomsdn.asp?TARGET=/xml/articles/hess061499.asp
and
http://web.archive.org/web/20000617215818/http://www.microsoft.com/isapi/gomsdn.asp?TARGET=/xml//default.asp