Unhandled Exceptions in SPSSDimensionNet Application Pool

Setup, update and manage desktop and/or server applications.

Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby ken_beckett » Fri May 16, 2008 5:09 pm

I submitted the following support request today to SPSS

At times we experience a high rate of failures with Site-To-Site Activations via DimensionNet. HTTPSYS is logging the following error originating from requests POSTed to ActivationWebService.asmx and FileManager.asmx: Connection_Abandoned_By_AppPool SPSSDimensionNetPool. Microsoft documentation states that the error "Connection_Abandoned_By_AppPool" indicates that the error was caused by a worker process stopping unexpectedly and orphaning requests.

Can you provide any insight on how to prevent these errors? Would increasing the number of worker processes in the SPSSDimensionNet application pool (i.e. configured as a web garden) offer any relief. Alternatively, would it be possible to have the ActivateWebService and the FileManagerWebService applications operate with their own dedicated application pool?


After futher investigation I updated the support case with the following:

After continued investigation, we've discovered the source of these Connection_Abandoned_By_AppPool SPSSDimensionNetPool errors. Each of these errors is preceded shortly by the following error found in the Application Event Log:

.NET Runtime 2.0 Error: EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 mrtables.webservice, P5 2.1.2.0, P6 4682617f, P7 129, P8 50, P9 exception, P10 NIL.

Additionally, each of these Connection_Abandoned_By_AppPool SPSSDimensionNetPool errors coincides to the second with the following warning found in the System Event Log:

The description for Event ID ( 1011 ) in Source ( W3SVC ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: SPSSDimensionNetPool, 4976,0

The number immediately following SPSSDimensionNetPool varies for each event logged. According to Microsoft KB911816 (http://support.microsoft.com/kb/911816), Unhandled exceptions cause ASP.NET-based applications to unexpectedly quit in the .NET Framework 2.0.

We plan to implement a custom UnhandledException Module to trap these errors. However, before we do, we seek input from SPSS on whether or not SPSS has experienced (and remedied) the same problems in their ASP practice either for the call center or for web interviewing.


I will share with the group anything I learn from SPSS. In the meantime I'm interested in knowing whether or not anyone else has experienced similar issues with DimensionNet on IIS 6.0 Win2K3
ken_beckett
 
Posts: 7
Joined: Fri May 16, 2008 1:29 pm

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby dporton » Mon May 19, 2008 4:18 am

I have been thinking about this , when you see this error , its a standard dot net 2 error

EventType clr20r3, P1 xxx, P2 xxx, P3 xxx, P4 xxx, P5 xxx, P6 xxx, P7 xxx, P8 xxx, P9 xxx, P10 xxx.

( i would like to know what the P1 .. Px etc means )

I have seen it in code I write ... it basically is telling you that you have not coded in all the exceptions and your program has spat the dummy .. because it does not know how to handle things … so for me … when I see this now , it is a generic dot net 2 error , so you will see lots as you program and refine your code …

now back to dimensions … this error

.NET Runtime 2.0 Error: EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 mrtables.webservice, P5 2.1.2.0, P6 4682617f, P7 129, P8 50, P9 exception, P10 NIL

from what I can see it talking about mrTables ? Is mrTables trashing something that stops other things ?

So some questions:

What version of dimensions are you using?
Are there mrTables users doing stuff when this happens?
Do the dimensions logs show any errors with mrTables?

And from a programmer’s point of few, you say

We plan to implement a custom UnhandledException Module to trap these errors.

How would you do this? Its not that I don’t believe you cannot do this, I would just like to know how you can, from a programers point of view?
dporton
 
Posts: 10
Joined: Wed Jan 30, 2008 4:08 pm
Location: Sydney Australia

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby ken_beckett » Mon May 19, 2008 9:33 am

While the error messages appear cryptic, I think that they provide some low level dump of process information at the time of the crash. That mrtables.webservice is listed is not necesarrily an indication that it was the source of the error.

You are correct about an unhandled exception but what is important to note is that according to KB911816 the unhandled exception is occurring outside the context of the request. In a ASP.NET 2.0 application, an unhandled exception (in that your code did not handle it) that happens inside the context of a request will be handled and wrapped in an HTTPExcpetion object. Its only when a exception is not handled outside the context of a request (for example when a separate thread is executed) that the exception will cause the worker process to end. (NOTE: I'm paraprahsing here. The KB article (http://support.microsoft.com/kb/911816) is quite detailed and describes our problem spot-on)

To answer your other questions:

What version of dimensions are you using? 4.0 (Soon to upgrade to 5.5)

Are there mrTables users doing stuff when this happens? Yes, but there also alot of other things going on with DimensionNet activities.

Do the dimensions logs show any errors with mrTables? No, but there aren't any useful errors coming from anything else either. The reason why? The worker process suddenly stops and gives no indication of what error occurred.

As far as implementing the UnhandledException Module, again please refer to the KB article. It provides detailed information on how to accomplish this.
ken_beckett
 
Posts: 7
Joined: Fri May 16, 2008 1:29 pm

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby ken_beckett » Mon May 19, 2008 2:35 pm

Update on message: EventType clr20r3, P1 xxx, P2 xxx, P3 xxx, P4 xxx, P5 xxx, P6 xxx, P7 xxx, P8 xxx, P9 xxx, P10 xxx.

Still haven't found documentation on what all this means, but what has become evident is that each and every one of the errors logged in the Application Event log that correspond to the SPSSDimensionNet application pool abandoning connections is this.

EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 mrtables.webservice, P5 2.1.2.0, P6 4682617f, P7 129, P8 50, P9 exception, P10 NIL.

To the best of my ability the Pn parametershave the following meaning...

P1, the hosting process, i.e. w3wp.e.e
P2, the hosting process version, 6.0.3790.1830
P3, ???
P4, the assembly from which the exception was raised, i.e. mrtables.webservice
P5, the assembly version
P6, ???
P7, ???
P8, ???
P9, the exception type raised
P10, ???

So, as dporton surmised, something in the mrtables.webservice assembly is causing the SPSSDimensionNet application pool to crash bringing down all other applications hosted by this pool including the Activate and FileManager WebServices.
ken_beckett
 
Posts: 7
Joined: Fri May 16, 2008 1:29 pm

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby ken_beckett » Mon May 19, 2008 6:38 pm

I think we've decided on abandoning the idea of registering an UnhandledExceptionModule as while testing it, I realized that the mrTables/WebService is logging to it's own DimensionNet log, TAB.tmp. When examining logs from last week, I came across some very telling information. Namely that each logged crash of the SPSSDimensionNet application pool occurs immediately after an exception is logged in the TAB.tmp log file.



I've annotated the logging to illustrate below. TAB prefixed entries originate from the TAB.tmp file. APPEVT entires originate from the Application Event log., HTTPSYS entries originate from the HttpErr log. Lastly, SYSEVT entires originate from the System Event Log.



Code: Select all
   TAB: 2008-05-14 09:36:39 errorServer: Value cannot be null. Parameter name: value
   TAB: 2008-05-14 09:36:39 --------------
   TAB: 2008-05-14 09:36:39 Stack trace:
   TAB: 2008-05-14 09:36:39 --------------
   TAB: 2008-05-14 09:36:39 >>>> Boolean SetProperties(String, String, String) File:  Line: 0
   TAB: 2008-05-14 09:36:39 >>>> System.Object _InvokeMethodFast(Object, Object[], SignatureStruct ByRef, Reflection.MethodAttributes, RuntimeTypeHandle) File:  Line: 0
   TAB: 2008-05-14 09:36:39 --- (stack dump stops at Invoke) ---
   TAB: 2008-05-14 09:36:39 Session ended (Void Session_End(Object, EventArgs), 0)
APPEVT: 2008-05-14 09:36:41 Type=Error, Source=.NET Runtime 2.0 Error Reporting, Description=EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 mrtables.webservice,
                            P5 2.1.2.0,   P6 4682617f, P7 129, P8 50, P9 exception, P10 NIL.
HTTPSYS: 2008-05-14 09:36:51 xxx.xx.x.xx yyyyyy xxx.xx.x.xx 80 HTTP/1.1 POST /SPSSMR/ActivateWebService/ActivateWebService.asmx - 1 Connection_Abandoned_By_AppPool SPSSDimensionNetPool
SYSEVT: 2008-05-14 09:36:51 Type=Warning, Source=W3SVC, Description=The following information is part of the event: SPSSDimensionNetPool...
   TAB: 2008-05-14 09:37:41 Web Service Started (Void Application_Start(System.Object, System.EventArgs), 0)


The APPEVT, HTTPSYS, and SYSEVT logs all indicate that there was a crash in the SPSSDimensionNet application pool. The very last entry shows that the mrTables/Web Service is starting back upon the first request received since the crash
ken_beckett
 
Posts: 7
Joined: Fri May 16, 2008 1:29 pm

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby swinstanley » Wed May 21, 2008 9:28 am

In my experience things work more smoothly if you isolate mrTables into its own process unless you are running under very light load, the primary reason for this is that you quite easily get into memory problems when its sharing a worker process with the rest of DimensionNet and this could be in part what you see here.

Regarding the technical details and the exceptions that are happening, much of this is un-managed code even though mrTablesWebService and the ActivateWebService are asmx's and that would suggest .Net they are all still accessing the Dimensions Datamodel which is outside the control of .Net and is also potentially outside the realms of what .Net 2.0 can manage in terms of exceptions too if things go very bad.

The reason why I'm pointing fingers at memory is that mrTables WebService consumes a lot, in big chunks mostly in the form of MDM documents but also some in terms of large MTD files etc.. Activation also consumes heaps of memory primarily because it needs to load multiple MDD files to merge them (and these are large). Are you monitoring memory usage on this process, if the virtual bytes gets near to 2GB it will crash out likely in a random, unmanaged place?

If you want to make a new AppPool, just copy the settings of the existing DimensionNet app pool in IIS Manager, then change the settings of the mrTables Front End and Web Service to use this new pool instead of the main DimensionNet pool.. I don't remember if there's DDL docs on doing this but there's not technical reason not to.

Cheers
Sam
Sam Winstanley - Director forgetdata
swinstanley at forgetdata.com
swinstanley
 
Posts: 22
Joined: Fri Jun 22, 2007 4:25 am
Location: London, UK

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby ken_beckett » Fri May 23, 2008 12:43 pm

Thanks for this. We will definitely take a look at memory usage in these processes.

About Application Pool and isolating activities. It is documented in the DDL (in 3 different articles) how to isolate and load balance DimensionNet and/or mrTables across multiple application pools. This does indeed sounds promising, but it leads me to wonder the following:

Is it necessary for every application available via the DimensionNet-Launcher application to run in the same application pool as DimensionNet-Launcher? Additionally, is it necessary to have the web service applications, e.g. ActivateWebSerivce, FileManagerWebService, MrTablesWebService run in the same application pool as DimensionNet-Launcher? My hunch was that perhaps the activities that are launched via DimensionNet-Launcher might need to be hosted in the same application pool simply because they may be accessing shared memroy. I also had thought that it would be unlikely that any web service would need to access shared memory and there fore did not need to be hosted in the same application pool. Your final thought gives me hope that there is actually no requirement for any DimensionNet application or web service to run in the same application pool. If this is indeed the case, then the only requirement is that these applications/services are hosted in application pools that are executing with the same identity as the SPSSDimensionNet pool.

I appreciate any insight you might of on this.

Thanks again for you input

Cheers
Ken
ken_beckett
 
Posts: 7
Joined: Fri May 16, 2008 1:29 pm

Re: Unhandled Exceptions in SPSSDimensionNet Application Pool

Postby swinstanley » Wed May 28, 2008 7:18 am

Sorry for the slow reply I was offline for some days. There's no technical requirement for the different apps to reside in the same app-pool and its quite okay for the heavier apps to be pulled out into their own AppPool. This has to be balanced with the fact that there's some physical overheads associated with each app-pool on your server in terms of the OS/IIS's management of extra resources for a new w3wp.exe etc.. if your server is used often as it sounds like it is then these are negligable.

Thanks
Sam
Sam Winstanley - Director forgetdata
swinstanley at forgetdata.com
swinstanley
 
Posts: 22
Joined: Fri Jun 22, 2007 4:25 am
Location: London, UK


Return to Installation/Infrastructure

Who is online

Users browsing this forum: No registered users and 1 guest

cron