isapi filter crashes after upgrade to sp2.2 on win2003

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

isapi filter crashes after upgrade to sp2.2 on win2003

Pavel Krejčí
Hello
I had fully working SP 2.1, I did full uninstall/restart/install from
the new msi package.
after restart the IIS worker process immediately started crashing
every time with the error in

app. log (not including the localized messages):
Application Failure  w3wp.exe 6.0.3790.3959 in msvcr90.dll 9.0.30729.1
at offset 00067892
and system log:
w3svc, event id 1011, data: 8007006d
w3svc, event id 1009, w3wp.exe process exit code: 0xc0000417

after replacing the shib. binaries with devel versions, i got this
error in popup:
---
Microsoft Visual C++ Debug Library : Debug Assertion Failed!

Program: c:\windows\system32\inetsrv\w3wp.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\write.c
Line: 68

Expression: (fh >= 0 && (unsigned)fh < (unsigned)_nhandle)

(Press Retry to debug the application)
---
I was unable to get further debugging info here. I dont have nor plan
to install VS2008. After trying with Debug Diagnostic Tools 1.1, the
errors moved to debugger itself (dbghost.exe crashing).

I managed to analyze the crash dump post-mortem from the non-debug
binaries with the following trace:

---
Function     Arg 1     Arg 2     Arg 3   Source
msvcr90!_write+59     ffffffff     0345f670     00000074
log4shib1_0!log4shib::FileAppender::_append+49     00000074
0000007f     be7d2c78
msvcp90!std::basic_string<char,std::char_traits<char>,std::allocator<char>
>::assign+6c     0345f930     03312757     011cdad8
0x011cdad8     011cdad8     011cdffc     00000000
log4shib1_0!log4shib::Category::callAppenders+6c     011cdad8
03406e40     03406df8
log4shib1_0!log4shib::Category::callAppenders+ba     011cdad8
011cdb6c     03406df8
log4shib1_0!log4shib::Category::callAppenders+ba     03459e98
00000001     0344e778
shibsp_lite1_2!shibsp::registerHandlers+514d     011ce068     0344e778
    78485ead
shibsp_lite1_2!shibsp::SimpleAttribute::clearSerializedValues+35dd
be7d0134     011ce6c4     01c6a038
shibsp_lite1_2!shibsp::SPConfig::instantiate+61b     033b9398
00000001     be7df16e
isapi_shib+908f     7c801c24     0000734c     011ce5b8
ntdll!RtlFreeHeap+70f     0009cb48     011ce900     00098968
ole32!CRpcChannelBuffer::SendReceive2+2fd     00098968     011cea34
 00000200
ole32!CAptRpcChnl::SendReceive+ab     00000000     011ce8a8     7763c552
0x000a0408     002b0928     011cec20     7c93a0d8
ntdll!RtlpAllocateFromHeapLookaside+13     7c93a11c     00000064
00000000
ntdll!RtlAllocateHeap+1dd     01c6a064     011cec48     6497f1a8
---

The configuration of the machine is:
Windows Server 2003 R2 Standard Ed., IIS 6.0, MSIE 8, php 5.2.9, mysql 5.
4-core xeon w/ HT, 4GB ram.

I've tested the new 2.2 release on WinXP SP3 + IIS 5.1 without
encountering this problem.
The shibd service started without problems and actually the error is
the same whether the service is running or not.

Any help greatly appreciated.

Greetings
Pavel Krejci
Reply | Threaded
Open this post in threaded view
|

RE: isapi filter crashes after upgrade to sp2.2 on win2003

Cantor, Scott E.
Pavel Krejčí wrote on 2009-06-18:
> Hello
> I had fully working SP 2.1, I did full uninstall/restart/install from
> the new msi package.
> after restart the IIS worker process immediately started crashing
> every time with the error in

99% of the time that's a permissions issue. The reason I advocate the
postinstall package for IIS installs is that it's less likely to reintroduce
permission errors to a working system. I'll try and make that point stronger
in the page.

> I managed to analyze the crash dump post-mortem from the non-debug
> binaries with the following trace:

That's in the logging code, which would indicate to me that it might not
have permissions to create or write to the native.log file. Check the rights
IIS has to var\log\shibboleth.

Of course, IIS runs under about 6 different identities seemingly based on
moon phase.

> The shibd service started without problems and actually the error is
> the same whether the service is running or not.

They aren't the same process, and shibd runs as local system in most cases,
which has broader file rights than IIS tends to.

Also, I had planned to look into trying to prevent the crash by reproducing
that and seeing if I could fix the logging code, but nobody filed a bug on
it, so it didn't get done.

-- Scott


Reply | Threaded
Open this post in threaded view
|

Re: isapi filter crashes after upgrade to sp2.2 on win2003

Pavel Krejčí
In reply to this post by Pavel Krejčí
2009/6/18 Scott Cantor <[hidden email]>:
>
> That's in the logging code, which would indicate to me that it might not
> have permissions to create or write to the native.log file. Check the rights
> IIS has to var\log\shibboleth.
>
> Of course, IIS runs under about 6 different identities seemingly based on
> moon phase.

Thanks for tip, that was right on spot.

IIS6 in native mode uses 3 identities by default:
- w3svc service - local system, which is unlikely to cause perms. problems
- w3wp worker proces - network service. It can be configured to
anything for any given applicaton (pool). shibboleth isapi filter runs
as part of this process.
- script engines (ie. php, everything configured in the extension
mappings) - they use impersonation by default, which means
IUSR_machinename when run as anonymous. I think the shibboleth isapi
dll runs as this when invoked as .sso handler. It can be also
configured in metabase to run w/o impersonation, ie. to use w3wp
identity (CreateProcessAsUser property).

In my case, the network service acc. somehow didnt have writing
permisions to native.log. Fixing that solved the problem.

Greetings
Pavel Krejčí
Reply | Threaded
Open this post in threaded view
|

RE: isapi filter crashes after upgrade to sp2.2 on win2003

Cantor, Scott E.
Pavel Krejčí wrote on 2009-06-19:
> In my case, the network service acc. somehow didnt have writing
> permisions to native.log. Fixing that solved the problem.

Could you please file a bug on this detailing how to reproduce the failure
(which permission was missing) so I can try and trap that crash?

It's not my code, but hopefully I can wrap the part that's failing, or worst
case I'll try and fix it in my fork of the logging code.

-- Scott