Showing posts with label WCF WebHost. Show all posts
Showing posts with label WCF WebHost. Show all posts

Friday, November 28, 2008

Cannot Find Server (404) or Get Plain Text for WCF .svc Files From IIS?

Question

After upgrading WCF (or NetFx-3.0) from an older beta version to RC1, I get “Page Not Found” (404) error or get the plain text content of the .svc file for .svc files. Is .svc extension removed from IIS?

Why?

If you installed WCF or NetFx-3.0 RC1 or later versions, you may get this problem. On Windows XP SP2, you may get the plain text of the .svc file. On Windows 2003 Server, you may get error like “Cannot find the page” (404) error. It looks like .svc extension is not supported in IIS.

Is .svc mapping removed from IIS? No. Definitely not! The reason why you saw this is because your IIS Metabase got messed up by an earlier WCF installer due to a nasty bug. You need to clean up the IIS Metabase manually so that it works again. How? Please keep on reading.

Before RC1, there was a bug in WCF installer. It accidentally populates .svc mapping to every path/node in IIS Metabase. This messed up the Metabase and has potential security concern. Fortunately, we found this issue in RC1 and fixed it. By fixing it, we only install .svc mapping to the template of the root /W3SVC of IIS Metabase. However, those machines with earlier WCF beta versions installed still have the messy ScriptMaps. They stop the inheritance of the .svc ScriptMaps from the template at the root /W3SVC. This is especially true for the “Default Web Site” and its virtual applications.

If you create a new web site with WCF RC1 installed, you won’t have this problem for that web site.

Solutions

Here are different ways that can help you to resolve the problem:

· Download tool: CleanIISScriptMaps.exe

Please find the tool “CleanIISScriptMaps.exe” in the attached zip file. The source for the tool is also included. You also find this tool and corresponding information on http://wcf.netfx3.com/files/folders/product_team/entry5648.aspx. The usage for this tool is simple, just run it without any argument from any command window. After running it, everything should just work fine. You don’t need to rerun any other setup.

If you find that .svc is still not supported for some virtual directories but you want them to be supported as well, you can run the tool with more options to explicitly remove the ScriptMaps properties from those paths. Here is the full usage of the tool:

Usage: CleanIISScriptMaps

Options:

By default, the tool removes redundant ScriptMaps properties.

/f Remove ScriptMaps from all nodes in IIS Metabase except for

the root "/W3SVC".

/p Prompt for confirmation when removing ScriptMaps at a node

that is different than that of the root "/W3SVC".

/v Print out verbose information.

/? Print this help.

· Reinstall IIS or re-create web sites

You can uninstall IIS and reinstall it so that the IIS Metabase is cleaned up. Then you will need to run the WCF install tool manually as following:

"%windir%\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe" /r /y

If you are running Windows 2003 Server, you may be able to delete the “Default Web Site” and re-create it. The problem will be fixed as well.

· Install .svc manually as a temporary workaround

You can run the following command to install .svc mapping manually. However, this does not clean up the mess of the IIS Metabase. You will see the same problem again when you install newer WCF. So this is the least recommended approach comparing to the above two options:

"%windir%\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe" /s:W3SVC

Though this does not clean up the Metabase for you, it does not mess up new web sites.

WinFX February CTP is released!

GREAT NEWS! The WinFX February CTP has just been released today and can be downloaded from here. Here is the readme document. This CTP has consumed a lot of customer feedback and thus brings quite a lot of breaking changes over the January one. The detailed WCF breaking changes can be found here. The changes include namespaces, APIs, configuration, setup, etc. Ed Pinto has a quick blog entry on this.

The biggest change to WebHost is that the .svc file format is changed. This was actually reflected in my earlier blog entries.

Before:

<%@Service language="C#" Debug="true"class="Microsoft.ServiceModel.Samples.CalculatorService" %>

After:

<%@ServiceHost language="C#" Debug="true"Service="Microsoft.ServiceModel.Samples.CalculatorService" %>

You can also use the factory model to create your own ServiceHost with the following advanced syntax:

<%@ServiceHost language="C#" Debug="true"Factory="Microsoft.ServiceModel.Samples.CalculatorServiceHostFactory" Service="Microsoft.ServiceModel.Samples.CalculatorService" %>

The Visual Studio CTP for this WinFX can be found at this link.

The WinFX SDK is also released as part of the larger Windows SDK which includes pre-release documentation, samples, and tools for both Win32® and WinFX® in Windows Vista.

Impersonation with Double Identities

There are two different levels of security that you can use for WCF services: Message level and Transport level. For Transport level, in HTTP case, you can use different IIS Authentication mechanisms to perform client authentication. If you do not carefully set the settings of the WCF binding, you may end up with both security levels enabled. This has a direct impact of impersonation. The reason is because WCF does not know which identity you want to apply to impersonation.

Generally, it is not recommended to have double securities enabled. However, there is actually a reason why people could get into this dilemma:

  • It is normal for people to disable anonymous access in IIS. In WCF, we want to make sure that a service explicitly specifies its transport level security settings. In this case, it means that you have to configure the service to not use anonymous authentication at transport layer. So when you also want to enable message-level security to perform message signing or encryption, you would get into this double-identity situation.

Here is a simple custom binding configuration that demonstrates how it is the case of double identities:

<customBinding>

<binding name="mixedSecureBinding">

<security/>

<httpTransport authenticationScheme="Ntlm"/>

binding>

customBinding>

It enables both message-level security and it also enables the NTLM authentication at the transport layer.

In this case, if you have an operation which requires impersonation, you would get the following error on the server side (you can see it from a debugger):

System.InvalidOperationException: Cannot start impersonation because the SecurityContext for the UltimateReceiver role from the request message with the 'http://www.microsoft.com/test/IHelloContract/Hello' action is not mapped to a Windows identity.

This is because WCF actually set the ServiceSecurityContext to WindowsIdentity.Anonymous when multiple identifies are found.

There are three different ways to resolve this issue.

1. Enable Anonymous Access from IIS

This is the simplest way, though there is a security concern for this approach. Here is the binding configuration that does not have the transport-layer authentication enabled:

<customBinding>

<binding name="mixedSecureBinding">

<security/>

<httpTransport/>

binding>

customBinding>

You have to enable the anonymous access in IIS to make this work, otherwise you would get ServiceActivationException on the server side due to mismatched settings between WCF binding and IIS.

This is not the ideal solution if you have other services in the same application that you want to have anonymous access disabled for security purpose.

2. Use Imperative Impersonation

When I said impersonation would fail above, I meant declarative (or automatic) impersonation. Here is the sample service operation that does this:

[OperationBehavior(Impersonation =ImpersonationOption.Required)]

public string Hello(string greeting)

To use imperative impersonation, you need to remove the “Impersonation” setting from the above declaration and perform impersonation from code. This requires having a Windows identity being retrieved from the context. Fortunately WCF provides a way for you to find the identities that are available in the stack:

OperationContext.Current.ServiceSecurityContext.AuthorizationContext.Properties["Identities"]

You can cast this into List and examine each identity of them. You cannot tell which identity is from which layer though. However, in WCF V1, the identity from the transport layer is always the first one. But this is guaranteed in future versions.

Once you have the Windows identity, you can perform the impersonation by yourself:

using (identity.Impersonate())

{

// Do something

}

3. Remove Unnecessary Identities Using Custom Authorization Policy

If you are still not satisfied by the above approaches, here is another choice for you. You need to use the ServiceAuthorizationBehavior.ExternalAuthorizationPolicies extension point to add an extra authorization policy to remove identities which are not needed. Here is a good sample about how to implement a custom authorization policy: http://msdn2.microsoft.com/en-gb/library/aa702720.aspx.

In our case, we can implement the Evaluate() method of the custom MyAuthorizationPolicy as following so that the last identity is picked:

public bool Evaluate(EvaluationContext evaluationContext, refobject state)

{

IList<IIdentity> identities = evaluationContext.Properties["Identities"] as IList<IIdentity>;

if (identities != null && identities.Count > 1)

{

// Pick the last one (which is likely to be the identity embeded in the message.

evaluationContext.Properties["Identities"] =

new List<IIdentity>(new IIdentity[] { identities[identities.Count - 1] });

}

return true;

}

Now you can configure the Authorization behavior in your ServiceHost. You need to implement ServiceHostFactory so that this custom behavior can be added to the ServiceHost. Here is an implementation of the CreateServiceHost method:

public override ServiceHostBase CreateServiceHost(stringconstructorString, Uri[] baseAddresses)

{

ServiceHost serviceHost = newServiceHost(typeof(HelloService), baseAddresses);

serviceHost.Authorization.ExternalAuthorizationPolicies =new ReadOnlyCollection<IAuthorizationPolicy>(

new IAuthorizationPolicy[] {

new MyAuthorizationPolicy()

});

return serviceHost;

}

The sample code for this blog entry is attached.

How to use .asmx extension to handle WCF requests?

For backward compatibility, WCF supports easy server-side migration from ASMX to WCF. This is good news for existing ASMX customers. Generally, the clients of ASMX may have server Urls hard-coded with the extension “.asmx”. For example, here is a sample Url:

http://www.myserver.com/foo/MyService.asmx

After installing WCF on the box, you can let WCF to handle the same client requests. Here are the steps about how to achieve this:

Replace BuildProvider for ASMX

You need to use the WCF service build provider for .asmx extension by seting the following in the web.config file of your virtual application:

<system.web>

<compilation>

<buildProviders>

<remove extension=".asmx" />

<add extension=".asmx" type="System.ServiceModel.Activation.ServiceBuildProvider, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />

buildProviders>

compilation>

system.web>

When WCF is running in the Mixed Transport mode (instead of AspNetCompatibility mode), WCF would intercept the request based on the BuildProvider information for the extension.

Replace HttpHandler for ASMX

If you are running WCF in AspNetCompatibility mode, you would also need to change the HttpHandler s for .asmx to the WCF HttpHandler. On IIS6, you just need to have something as following in your config:

<system.web>

<httpHandlers>

<remove path="*.asmx" verb="*">remove>

<add path="*.asmx" verb="*" type="System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"

validate="false" />

httpHandlers>

system.web>

On IIS7, things are different. IIS7 introduces two different HTTP pipelines: Integrated and Classic. The latter works similar as IIS6 in the way that all of the managed requests go through the ASP.NET ISAPI filter. When this happens, the HttpModules and HttpHandlers defined in web.config files would be invoked for the corresponding requests.

When the virtual application is running in the Integrated mode, however, the modules and handlers defined in web.config files under the section are totally ignored. All of the requests are handled straightforward by IIS7. So you need to make sure that you have integrated handlers defined for the corresponding extensions. We can set the following in the web.config of the virtual application to achieve this:

<system.webServer>

<handlers>

<remove name="WebServiceHandlerFactory-Integrated"/>

<add name="MyNewAsmxHandler" path="*.asmx" verb="*"type="System.ServiceModel.Activation.HttpHandler, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"preCondition="integratedMode,runtimeVersionv2.0" />

handlers>

<validation validateIntegratedModeConfiguration="false" />

system.webServer>

The section is actually rooted in %windir%\system32\inetsrv\config\applicationhost.config instead of in the root web.config in %windir%\Microsoft.Net\Framework{64}\v2.0.50727\config. The element takes the “name” as the key instead of the “path” as we did to the section. You must specify “validateIntegratedModeConfiguration” to false to make IIS happy.

Correct SOAP Actions

Once you configured above settings, you can create a .asmx file which looks exactly as a .svc file. For example, you can have the following content:

<% @ServiceHost language = "c#" Factory="HelloWorld.SimpleServiceHostFactory" %>

Now you can browse your service with an .asmx extension. At this point, all requests will go through happily into WCF channel stack.

One last thing that you need to be aware is that you need to make sure your WCF service is backward compatible to ASMX so that ASMX clients can communicate with it. Basically you need to use the interoperable binding such as BasicHttpBinding. And you need to make sure the request/response SOAP actions are expected for ASMX clients. Here is an example:

[ServiceContract(Name = "RequestReplyService", Namespace ="http://myoldasmxworld.org/")]

public interface IRequestReplySyncServiceAsmx

{

[OperationContract(IsOneWay = false, Action ="http://myoldasmxworld.org/HelloRequestReply")]

string HelloRequestReply(string text);

}

It is compatible to the following ASMX web service:

[WebService(Name = "MyAsmxService", Namespace ="http://myoldasmxworld.org/")]

public class AsmxRequestReplyService

{

[WebMethod]

public string HelloRequestReply(string text)

{

// Implementation

}

}

How to redirect a Url in IIS6/IIS7 and in ASP.NET?

Sometimes you want to redirect users request to a different one when you want to retire old ones but still provide backward-compatibility. David Wang has a good summary on different redirects for IIS in his blog entry. Here I just want to add a few points which have helped people to resolve tricky problems.

First of all, as David pointed out, there are two (or three in David’s terms) types of redirects:

· Client-side redirection: the server sends a “302 Redirect” response to the client together with the new location header.

· Server-side redirection: the server rewrites the request with another URL transparently and the client is not aware of anything.

Client-side redirection

It is also called HTTP redirection. You can achieve it in the following ways.

Redirect in IIS6

You can use the HttpRedirect Metabase setting to perform redirection in IIS6. This property can be set to a virtual application, a file, or a virtual directory (a.k.a. web folder). It requires the physical existence of these “source” items. To achieve that, you can use IIS Manager. Here are the steps on how to set it for a file:

· Right click on the file and click on Properties

· Click on “File” tab of the dialog

· Select “A redirection to a URL” radio box

· Type in your new Url, and/or make any other changes, and click OK.

After this, you would have setup the file so that all requests to this file is redirected to the new Url. You can check the Metabase setting with the following command:

cscript.exe %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs get /w3svc/1/Root/foo/p1.aspx/HttpRedirect

If the file does not have any Metabase property pre-defined in the Metabase, you cannot use adsutil.vbs to set this property. But you can set it with the script to a virtual application.

Redirect in IIS7

In IIS7, the concept is similar as that of IIS6. You can use the new command “appcmd.exe” to enable redirection as following:

%windir%\system32\inetsrv\appcmd set config "Default Web Site/foo" -section:system.webServer/httpRedirect -enabled:true -destination:"http://localhost/bar/p1.aspx"

This tells IIS to redirect all request sending to the virtual application “/foo” to “/bar/p1.aspx”. The actual result is that appcmd.exe adds the following section to the web.config file for the application “/foo”:

<system.webServer>

<httpRedirect enabled="true"destination="http://localhost/bar/bbb.aspx" />

system.webServer>

Redirect with ASP.NET

You can also use ASP.NET to implement client-side redirection. If you don’t want ASP.NET pipeline to intercept any such request, you would need to implement the logic in the global.asax file. This is important for WCF applications since WCF intercepts requests from the pipeline for WCF requests. Here is some sample code for a global.asax file:

<%@ Application Language="C#" Debug="true" %>

This approach is more flexible than those provided by IIS:

· It does not require the existence of the physical file.

· It is imperative and thus you can add any custom programming logic to determine whether you want to redirect the request. For example, you can check the query string to see whether it contains the information that requires a redirection.

Server-side redirection

It is sometimes also called Url rewriting or Url aliasing. One of the advantages for server-side redirection is performance because there is no round-trip involved if the redirection is on the same machine.

Redirect with ASP.NET

You can use ASP.NET to implement the server-side redirection with Server.Transfer(path) method.

WCF Request Throttling and Server Scalability

Two Threads per Request

In .NET 3.0 and 3.5, there is a special behavior that you would observe for IIS-hosted WCF services. Whenever a request comes in, the system would use two threads to process the request:

· One thread is the CLR ThreadPool thread which is the worker thread that comes from ASP.NET.

· Another thread is an I/O thread that is managed by the WCF IOThreadScheduler (actually created by ThreadPool.UnsafeQueueNativeOverlapped).

When you have a high latency request, you would see the following callstack in the debugger before the request is completed (with .NET 3.5):

0ee6ee8c 5094dad5 mscorlib_ni!System.Threading.WaitHandle.WaitOne()+0xa

0ee6ee8c 50951e3b System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.ExecuteSynchronous(System.Web.HttpApplication, Boolean)+0x8d

0ee6eeb4 65fe626d System_ServiceModel_ni!System.ServiceModel.Activation.HttpModule.ProcessRequest(System.Object, System.EventArgs)+0x143

0214e4f4 65fe3fd1 System_Web_ni!System.Web.HttpApplication+SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+0x5d

0ee6ef08 65fe804f System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+0x41

0ee6efb8 65fe4df2 System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+0x63b

060f05dc 66003a92 System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+0x56

0ee6f040 65fd8022 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+0x352

0ee6f0c4 65fd7e07 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x1f2

0ee6f0f4 01782374 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+0x1b

0ee6f138 6a2abf3c webengine!MgdGetCurrentNotification+0x236

0ee6f19c 74a22ea0 webengine!MgdGetPreloadedSize+0x4d

0ee6f1b0 74a23696 iiscore+0x2ea0

0ee6f560 6a2ac222 webengine!MgdCanDisposeManagedContext+0xb5

0ee6f570 015f1311 webengine!MgdIndicateCompletion+0x22

System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+0x289

This is the worker thread that comes from ASP.NET. Another thread is still processing the request in WCF:

01fbf0ac 0e9001c7 App_Web_rsfhd6l1!HelloWorld.SimpleService.Hello(System.String)+0x14

01fbf0ac 50b8d90b System_ServiceModel_ni!DynamicClass.SyncInvokeHello(System.Object, System.Object[], System.Object[])+0x3f

01fbf0ac 50b6d245 System_ServiceModel_ni!System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(System.Object, System.Object[], System.Object[] ByRef)+0x1fb

01fbf108 509137ad System_ServiceModel_ni!System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0xd5

01fbf148 509136a6 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0xad

01fbf174 50913613 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x76

062b75a8 50913459 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x33

062b75a8 50912257 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x59

062b75a8 50911f8f System_ServiceModel_ni!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(System.ServiceModel.Dispatcher.MessageRpc ByRef)+0x127

01fbf1f0 509115ff System_ServiceModel_ni!System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)+0xdf

01fbf394 5090f8c9 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext, Boolean, System.ServiceModel.OperationContext)+0x1ff

01fbf3dc 5090f35e System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext, System.ServiceModel.OperationContext)+0x169

01fbf42c 5090f2f1 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult)+0x5e

01fbf42c 50232d68 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult)+0x41

01fbf42c 50904501 SMDiagnostics_ni!System.ServiceModel.Diagnostics.Utility+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)+0x28

01fbf468 50992b36 System_ServiceModel_ni!System.ServiceModel.AsyncResult.Complete(Boolean)+0xb1

01fbf4d8 50992215 System_ServiceModel_ni!System.ServiceModel.Channels.InputQueue`1+AsyncQueueReader[[System.__Canon, mscorlib]].Set(Item)+0x46

01fbf4d8 50991ffb System_ServiceModel_ni!System.ServiceModel.Channels.InputQueue`1[[System.__Canon, mscorlib]].EnqueueAndDispatch(Item, Boolean)+0x1f5

00000000 5091d7e5 System_ServiceModel_ni!System.ServiceModel.Channels.InputQueue`1[[System.__Canon, mscorlib]].EnqueueAndDispatch(System.__Canon, System.ServiceModel.Channels.ItemDequeuedCallback, Boolean)+0x6b

00000001 50977b7e System_ServiceModel_ni!System.ServiceModel.Channels.SingletonChannelAcceptor`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Enqueue(System.__Canon, System.ServiceModel.Channels.ItemDequeuedCallback, Boolean)+0x75

01fbf570 5094f396 System_ServiceModel_ni!System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(System.ServiceModel.Channels.HttpRequestContext, System.ServiceModel.Channels.ItemDequeuedCallback)+0x1ce

01fbf5b8 5094e4cf System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(System.ServiceModel.Activation.HostedHttpRequestAsyncResult)+0xc6

0216803c 5094defd System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()+0xdb

01fbf60c 5094dea5 System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()+0x19

01fbf638 50903be6 System_ServiceModel_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)+0x29

01fbf674 50903b26 System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke2()+0x36

01fbf688 50903ab5 System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke()+0x4a

01fbf6bc 5090390f System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.ProcessCallbacks()+0x185

01fbf6e8 5090388b System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.CompletionCallback(System.Object)+0x6f

01fbf720 50232e1f System_ServiceModel_ni!System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0xf

01fbf720 79405534 SMDiagnostics_ni!System.ServiceModel.Diagnostics.Utility+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x2f

01fbf744 79e7c74b mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x60

01fbf758 79e7c6cc mscorwks!LogHelp_TerminateOnAssert+0x3433

01fbf7d8 79f00eca mscorwks!LogHelp_TerminateOnAssert+0x33b4

01fbf7f8 79f00e75 mscorwks!GetCLRFunction+0x1de16

Why is this? Is this a scalability issue for WCF? I will explain this below.

Asynchronous Request Processing

One of the most significant pieces for WCF is that it is highly asynchronous from bottom to top for the whole channel stack and the programming model. The advantage is apparent: applications have high flexibility by issuing operations asynchronously, doing some other stuff in parallel, and coming back to handle the completed operations once they are completed. This is also scalable when the asynchronous operations are I/O bound so that you do not need to hold extra threads to wait for operation completions.

ASP.NET also supports asynchronous request processing mechanisms through IHttpModule and IHttpAsyncHandler. With these interfaces, ASP.NET can push many concurrent requests up to the layers which have such asynchronous implementations. In this way, the whole stack need a small number of threads to process a large number of concurrent requests given appropriate queuing for the requests.

However, this also exposes a memory problem when this is on IIS 6.0. When there is unlimited number of incoming requests, all of the requests would be pushed up by ASP.NET to upper layers and it would cause significant memory consumption. Thus it would cause the server to be unavailable to do its work correctly.

So in the release of .Net 3.0 and 3.5, WCF implemented synchronous versions of HTTP module and handler instead of asynchronous ones. See the following for more details.

ASP.NET Threading on IIS 6.0

When ASP.NET is hosted in IIS 6.0, requests are handed over to ASP.NET on IIS I/O threads. ASP.NET uses CLR ThreadPool worker threads to handle requests. The CLR ThreadPool automatically adjusts the number of threads based on the server workload. In order to prevent from using too many threads or using too much memory, ASP.NET has a limit on the number of threads concurrently executing requests. This is controlled by the httpRuntime/minFreeThreads and httpRuntime/minLocalRequestFreeThreads settings. If the limit is exceeded, the request is queued in the application-level queue, and executed later when the concurrency falls back down below the limit. Thomas Marquardt has written an excellent blog on this:

http://blogs.msdn.com/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx (TM0707)

In .NET 2.0, the magic config “processModel/autoConfig” in machine.config automatically configures the ASP.NET threading settings for most applications. For those which require special settings need to be configured differently. The following KB article was provided as the general guidance:

http://support.microsoft.com/kb/821268 (KB821268)

However, there is no throttling for concurrent requests in ASP.NET on IIS 6.0.

WCF Request Throttling and Tuning

Because of lacking the throttling for concurrent requests in ASP.NET, WCF would get unlimited concurrent requests pushed up from ASP.NET if WCF had implemented asynchronous HTTP module/handler on IIS 6.0. It would cause severe memory usage problem. Because of this concern, WCF implemented the synchronous HTTP module/handler in 3.0 and 3.5 instead.

Here I only describe how the HTTP module works for WCF. The logic for the HTTP handler is similar. When a request comes in from ASP.NET, WCF grabs the request if it is a WCF request (by checking the registered BuildProvider of the extension to see whether it is WCF service BuildProvider). After that, it immediately switches to a new I/O thread and hold the ASP.NET worker thread until the request is completed. You may ask, why doesn’t WCF reuse the ASP.NET worker thread to handle the request until it is completed? Here are the reasons:

· As long as the ASP.NET worker thread is not returned back to ASP.NET, the request is not able to be completed. The client would be blocked.

· When the WCF operation is one-way, the request has to be completed before the service operation is invoked. So the ASP.NET worker thread has to return earlier.

· WCF supports people to complete a two-way operation earlier in a service operation when you call RequestContext.Close.

All of these require that WCF should not hold the ASP.NET worker thread once the request is claimed to be “completed”. At the same time, WCF has to hold the ASP.NET worker thread in a waited state before the request is completed due to the requirement of synchronous request processing logic. WCF relies on this ASP.NET thread throttling logic to limit concurrent requests.

Intuitively this seems to have scalability issue. Fortunately for most WCF services, especially high-throughput ones, this has quite low impact as long as the server is busy processing requests. This is demonstrated as the CPU usage of the server.

On multi-proc/multi-core servers, the default limit of concurrent requests may not be enough for the server to achieve best throughput. This would require larger limits for ASP.NET worker threads to be used to handle requests.

Throttling Tuning

In order to achieve high throughput on high-end servers, you would need to make the following throttling tuning:

· Increase ASP.NET thread throttling to allow more concurrent worker threads to handle requests

· Increase WCF service throttling to allow more concurrent requests

The latter reflects the change of the known WCF throttling settings such asMaxConcurrentCalls, MaxConcurrentInstances, and MaxConcurrentSessions. The former would require fine tuning of ASP.NET thread throttling as documented in the KB article KB821268 mentioned above. Basically ASP.NET does not execute more than the following number of concurrent requests:

(maxWorkerThreads*number of CPUs)-minFreeThreads

This is 12 by default on a single CPU machine. In order to get more threads to process requests, you need to increase maxWorkerThreads. Once you can get CPU fully loaded with the settings, setting “minWorkerThreads” to 2 or 3 would allow WCF to have best throughput. Here is sample setting in machine.config:

The above settings work for most WCF applications. However, when the service has very slow operations which are I/O bound (for example, they further connects to backend database or network or file system to perform high-latency operations), you would need to use many outstanding ASP.NET worker threads to achieve parallel requests. This is bad when the number of requests is huge, for example, several thousands or more. Each thread would hold a fair amount of system resource, i.e., memory. It would cause the service to be very slow to respond. This really requires a real asynchronous design to solve the problem. Because of this, ASP.NET introduced a new throttling logic for concurrent requests in IIS 7.0 to support this scenario for WCF.

ASP.NET Throttling on IIS 7.0 in Integrated Mode

As Thomas mentioned in his blog entry (TM0707) mentioned above, the following registry value is introduced to throttle concurrent requests:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0]

“MaxConcurrentRequestsPerCpu”=dword:0000000c

The default value is 12 per CPU which means is similar as the default setting for ASP.NET worker threads. This was introduced in ASP.NET 2.0 SP1 (shipped with .NET 3.5) and thus it is available in Windows 2008. It only applies to the IIS 7.0 Integrated mode, which is exactly where WCF is registered on Windows 2008.

Scalable Solutions for High Latency Services

Though the above throttling support was added to Windows 2008, WCF still uses synchronous module/handler instead of asynchronous ones in the released versions due to timing constraint. However, the internal WCF design does allow asynchronous request processing and it is just one step away from the real asynchronous logic. Here I provide a simple wrapper for that.

Custom HttpModule Using Reflection

The internal WCF types support asynchronous HTTP module. Since they are internal, we have to rely on reflection to create a real one.

First of all, you would need to register the async events for your HttpModule as following:

context.AddOnPostAuthenticateRequestAsync(

MyHttpModule.beginEventHandler,

MyHttpModule.endEventHandler);

This is registered to the PostAuthenticateRequest for the same reasons as the default WCF HttpModule.

Secondly, you would need to use reflection to create the instance of the internal WCF type System.ServiceModel.Activation.HostedHttpRequestAsyncResult:

Type hostedHttpRequestAsyncResultType =typeof(ServiceHostingEnvironment).Assembly.GetType("System.ServiceModel.Activation.HostedHttpRequestAsyncResult");

ConstructorInfo hostedHttpRequestAsyncResult = hostedHttpRequestAsyncResultType.GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Public,

null, new Type[] { typeof(HttpApplication), typeof(bool),typeof(AsyncCallback), typeof(object) }, null);

return (IAsyncResult)hostedHttpRequestAsyncResult.Invoke(new object[] { application, false, cb, extraData });

Once you have the custom HttpModule, you can replace it with the default one that is registered in the WAS configuration %windir%\system32\inetsrv\config\applicationhost.config:

<system.webServer>

<modules>

<add name="ServiceModel"type="System.ServiceModel.Activation.HttpModule, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"preCondition="managedHandler" />

<modules>

<system.webServer>

To do that, you can also define local settings in web.config:

<system.webServer>

<modules>

<remove name="ServiceModel"/>

<add name="MyAsyncWCFHttpModule" type="AsyncWcfModule.MyHttpModule, AsyncWcfModule" preCondition="managedHandler" />

modules>

<validation validateIntegratedModeConfiguration="false" />

system.webServer>

The sample code is attached.

Improvement in .NET 3.5 SP1

In .NET 3.5 SP1, we will provide the real asynchronous HTTP module/handler for WCF to better solve this problem.