SP Crashing

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

SP Crashing

Dan McLaughlin
We are running Apache/2.4.37 (Win64) OpenSSL/1.1.1 mod_jk/1.2.46 on
Windows 2008 SP2 and Shibboleth 3.0.4.   Over the past few months
we've started experiencing issues with the SP crashing during peak
load.  We see it crash in the mornings and afternoons before 5PM. We
will usually peak at about 1000-1200 logged in users with concurrent
requests approaching 200 in the mornings and slightly lower in the
afternoons.

The shibd.exe application fault has occurred in conjunction with
module "unknown", "ntdll", and "libcrypto" as shown below.

Are there memory tuning settings in the SP and or Apache that we
should be looking at?  Are there any best practices for matching up
Apache tuning with SP tuning related to threads, timeouts, etc... that
would should make sure are in sync?  Is there any SP logging that we
could enable that you might think would help narrow in on the root
cause?

Faulting application name: shibd.exe, version: 3.0.4.0, time stamp: 0x5c81b93a
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000000
Faulting process id: 0x5ee4
Faulting application start time: 0x01d56cc5ddbccdd9
Faulting application path: E:\servers\shib-sp\sbin64\shibd.exe
Faulting module path: unknown
Report Id: 5b3b80ab-d95e-11e9-bb81-005056010496

Fault bucket , type 0
Event Name: BEX64
Response: Not available
Cab Id: 0

Faulting application name: shibd.exe, version: 3.0.4.0, time stamp: 0x5c81b93a
Faulting module name: libcrypto-1_1_1-x64.dll, version: 1.1.1.1, time
stamp: 0x5c0f2b49
Exception code: 0xc0000005
Fault offset: 0x00000000001783a5
Faulting process id: 0x5cc8
Faulting application start time: 0x01d56d6b4451897b
Faulting application path: E:\apps\shib-sp\sbin64\shibd.exe
Faulting module path: C:\Program Files\Shibboleth\SP\lib\libcrypto-1_1_1-x64.dll
Report Id: 8469cccb-d95e-11e9-bb81-005056010496

Faulting application name: shibd.exe, version: 3.0.4.0, time stamp: 0x5c81b93a
Faulting module name: ntdll.dll, version: 6.1.7601.24511, time stamp: 0x5d3fa9bd
Exception code: 0xc0000005
Fault offset: 0x000000000002a365
Faulting process id: 0x4f3c
Faulting application start time: 0x01d56d6b6b702f7b
Faulting application path: E:\apps\shib-sp\sbin64\shibd.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: eb2b6d4a-d989-11e9-bb81-005056010496


--

Thanks,

Dan
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
On 9/19/19, 1:33 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:

> The shibd.exe application fault has occurred in conjunction with
> module "unknown", "ntdll", and "libcrypto" as shown below.

I would be monitoring memory size and seeing how much it spikes but under Windows I have load tested plenty of times in the old days and pushed it very far without any problems apart from slowness. If I had to guess, you're perhaps using features that haven't been adequately tested and hitting bugs in the system that postdate a lot of that older testing and shaking out. Or there are regressions in V3.
 
> Are there memory tuning settings in the SP and or Apache that we
> should be looking at?

Just avoiding prefork on Unix. The stack size should be much less than the default is, but it doesn't really matter as long as prefork isn't used. The thread count shouldn't ever get very large as long as the Apache thread count is reasonable. The logs I think will prefix the records with a thread ID so keeping an eye on how many there are is another data point. It would take a lot to cause problems, but with prefork you get 200+, and that will break.

Otherwise shibd doesn't need any tuning, apart from expiring sessions more often to reduce the running footprint if that's really a problem. On Windows, the size will shrink back down in concert with usage, vs. Linux where the peak memory usage will stay allocated to the process and never go down.

> Is there any SP logging that we could enable that you might think would help narrow in on the root
> cause?

The more the better, but the SP won't run well under load on DEBUG, it just grinds to a halt, or used to.

I don't really know how to debug anything post-mortem on Windows, get symbols in place, etc. On Linux the core dump under gdb frequently provides hints that tend to point at the right culprit. I ironically do all my development debugging on Windows, but all my crash research on Linux.

The fact is that crashes all over the place with no consistent stack trace really screams out of memory, that's just the one constant that can bite any code anywhere with no repeatability.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
On 9/19/19, 1:55 PM, "Cantor, Scott" <[hidden email]> wrote:

> Just avoiding prefork on Unix. The stack size should be much less than the default is, but it doesn't really matter as
> long as prefork isn't used. The thread count shouldn't ever get very large as long as the Apache thread count is
> reasonable.

I should perhaps not be so blasé about this, since I don't have any history with Windows usage and how it might behave under real world high loads. The default stack is, I think 1M, which is still nuts, and can certainly be drastically lowered and is a good simple step to take (it's documented how to lower that). If load spikes are for some reason not getting in and out of shibd fast enough to keep the worker thread count inside it from spiking, that definitely breaks on Linux and I suppose could break on Windows. But the size is bigger on Linux, so that's one reason the spikes are more fatal there.

But all my experience is that shibd is fast enough doing its work that as long as there aren't 100 processes connecting to it, the thread size stays manageable. Each process pools client sockets to connect, but with 100 web processes, you get a minimum of 100 threads in shibd.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
In reply to this post by Cantor, Scott E.
I have the debug symbols loaded and SP source downloaded with Windbg
installed.  Here's what I've found out so far about the two crashes
this morning, unfortunately we have apps that come grab all the WER
reports daily, so I don't have access to the dumps from the other
crashes at this time.  Is there any logging I could enable that might
help point me in the right direction?

shibd.exe Stopped working 9/19/2019 8:31:17 AM spservice 0xc0000005
Access Violation 0x0002a365 ntdll.dll 6.1.7601.24511
E:\servers\shib-sp\sbin64\shibd.exe 12,226

shibd.exe Stopped working 9/19/2019 10:21:50 AM spservice 0xc0000374 A
heap has been corrupted.
 0x000bf302 StackHash_1aea 6.1.7601.24511

--

Thanks,

Dan

On Thu, Sep 19, 2019 at 12:55 PM Cantor, Scott <[hidden email]> wrote:

>
> On 9/19/19, 1:33 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:
>
> > The shibd.exe application fault has occurred in conjunction with
> > module "unknown", "ntdll", and "libcrypto" as shown below.
>
> I would be monitoring memory size and seeing how much it spikes but under Windows I have load tested plenty of times in the old days and pushed it very far without any problems apart from slowness. If I had to guess, you're perhaps using features that haven't been adequately tested and hitting bugs in the system that postdate a lot of that older testing and shaking out. Or there are regressions in V3.
>
> > Are there memory tuning settings in the SP and or Apache that we
> > should be looking at?
>
> Just avoiding prefork on Unix. The stack size should be much less than the default is, but it doesn't really matter as long as prefork isn't used. The thread count shouldn't ever get very large as long as the Apache thread count is reasonable. The logs I think will prefix the records with a thread ID so keeping an eye on how many there are is another data point. It would take a lot to cause problems, but with prefork you get 200+, and that will break.
>
> Otherwise shibd doesn't need any tuning, apart from expiring sessions more often to reduce the running footprint if that's really a problem. On Windows, the size will shrink back down in concert with usage, vs. Linux where the peak memory usage will stay allocated to the process and never go down.
>
> > Is there any SP logging that we could enable that you might think would help narrow in on the root
> > cause?
>
> The more the better, but the SP won't run well under load on DEBUG, it just grinds to a halt, or used to.
>
> I don't really know how to debug anything post-mortem on Windows, get symbols in place, etc. On Linux the core dump under gdb frequently provides hints that tend to point at the right culprit. I ironically do all my development debugging on Windows, but all my crash research on Linux.
>
> The fact is that crashes all over the place with no consistent stack trace really screams out of memory, that's just the one constant that can bite any code anywhere with no repeatability.
>
> -- Scott
>
>
> --
> To unsubscribe from this list send an email to [hidden email]
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
Where is the debug version of the shibd.exe?  It use to be under a
./debug folder, but it's not there anymore.  Do you have the debug
version or symbols available for download somewhere?

--

Thanks,

Dan

On Thu, Sep 19, 2019 at 4:37 PM Dan McLaughlin
<[hidden email]> wrote:

>
> I have the debug symbols loaded and SP source downloaded with Windbg
> installed.  Here's what I've found out so far about the two crashes
> this morning, unfortunately we have apps that come grab all the WER
> reports daily, so I don't have access to the dumps from the other
> crashes at this time.  Is there any logging I could enable that might
> help point me in the right direction?
>
> shibd.exe Stopped working 9/19/2019 8:31:17 AM spservice 0xc0000005
> Access Violation 0x0002a365 ntdll.dll 6.1.7601.24511
> E:\servers\shib-sp\sbin64\shibd.exe 12,226
>
> shibd.exe Stopped working 9/19/2019 10:21:50 AM spservice 0xc0000374 A
> heap has been corrupted.
>  0x000bf302 StackHash_1aea 6.1.7601.24511
>
> --
>
> Thanks,
>
> Dan
>
> On Thu, Sep 19, 2019 at 12:55 PM Cantor, Scott <[hidden email]> wrote:
> >
> > On 9/19/19, 1:33 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:
> >
> > > The shibd.exe application fault has occurred in conjunction with
> > > module "unknown", "ntdll", and "libcrypto" as shown below.
> >
> > I would be monitoring memory size and seeing how much it spikes but under Windows I have load tested plenty of times in the old days and pushed it very far without any problems apart from slowness. If I had to guess, you're perhaps using features that haven't been adequately tested and hitting bugs in the system that postdate a lot of that older testing and shaking out. Or there are regressions in V3.
> >
> > > Are there memory tuning settings in the SP and or Apache that we
> > > should be looking at?
> >
> > Just avoiding prefork on Unix. The stack size should be much less than the default is, but it doesn't really matter as long as prefork isn't used. The thread count shouldn't ever get very large as long as the Apache thread count is reasonable. The logs I think will prefix the records with a thread ID so keeping an eye on how many there are is another data point. It would take a lot to cause problems, but with prefork you get 200+, and that will break.
> >
> > Otherwise shibd doesn't need any tuning, apart from expiring sessions more often to reduce the running footprint if that's really a problem. On Windows, the size will shrink back down in concert with usage, vs. Linux where the peak memory usage will stay allocated to the process and never go down.
> >
> > > Is there any SP logging that we could enable that you might think would help narrow in on the root
> > > cause?
> >
> > The more the better, but the SP won't run well under load on DEBUG, it just grinds to a halt, or used to.
> >
> > I don't really know how to debug anything post-mortem on Windows, get symbols in place, etc. On Linux the core dump under gdb frequently provides hints that tend to point at the right culprit. I ironically do all my development debugging on Windows, but all my crash research on Linux.
> >
> > The fact is that crashes all over the place with no consistent stack trace really screams out of memory, that's just the one constant that can bite any code anywhere with no repeatability.
> >
> > -- Scott
> >
> >
> > --
> > To unsubscribe from this list send an email to [hidden email]
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
In reply to this post by Dan McLaughlin
On 9/19/19, 5:38 PM, "Dan McLaughlin" <[hidden email]> wrote:

> I have the debug symbols loaded and SP source downloaded with Windbg
> installed.

The result is pretty indistinguishable to me from not having them, so I'd have to take your word for it, I don't actually know what that would look like. The heap corruption does suggest a real bug and not just memory allocation problems, but the only bugs that can usually be tracked down are the ones that are reproducible with specific transactions.

> Is there any logging I could enable that might help point me in the right direction?

As I said, there's DEBUG and there's nothing, and DEBUG itself is pretty much nothing with this sort of thing unless it's reproducible under specific circumstances with a single transaction, otherwise it's just noise. Race conditions don't manifest anywhere near the actual code causing them.

INFO alone would log metadata or configuration reload events, and if there are any in proximity to the crashes every time that might be suggestive.

If you're using any newer V3 features, I'd avoid them and if that makes a difference then that's a solid direction for research, but nothing that's going to result in a fast answer.

If there really is a race condition (and since it's not crashing for lots of other people across thousands of installs, it seems plausible that it's load-based), then spreading the load is obviously the most likely thing that will reduce the frequency.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
In reply to this post by Dan McLaughlin
On 9/19/19, 7:02 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:

> Where is the debug version of the shibd.exe?  It use to be under a
> ./debug folder, but it's not there anymore.  Do you have the debug
> version or symbols available for download somewhere?

No. I can zip up a sandbox build with all the files in the next few days. I don't think there have been any code commits since the last release so it would match.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
Sounds good, in the meantime, I'm trying to see if I can build it
myself from the zip file I downloaded.  I've never built it before, so
I'm assuming that's enough.  This is when a docker build image would
be a nice to have.

--

Thanks,

Dan McLaughlin
Technology Consortium, LLC
[hidden email]
mobile: 512.633.8086
http://www.tech-consortium.com

NOTICE: This e-mail message and all attachments transmitted with it
are for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is strictly prohibited. The contents of
this e-mail are confidential and may be subject to work product
privileges. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message.


On Thu, Sep 19, 2019 at 6:10 PM Cantor, Scott <[hidden email]> wrote:

>
> On 9/19/19, 7:02 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:
>
> > Where is the debug version of the shibd.exe?  It use to be under a
> > ./debug folder, but it's not there anymore.  Do you have the debug
> > version or symbols available for download somewhere?
>
> No. I can zip up a sandbox build with all the files in the next few days. I don't think there have been any code commits since the last release so it would match.
>
> -- Scott
>
>
> --
> To unsubscribe from this list send an email to [hidden email]
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP Crashing

Rod Widdowson
> Sounds good, in the meantime, I'm trying to see if I can build it
> myself from the zip file I downloaded.  I've never built it before, so
> I'm assuming that's enough.  

If you are not aware of it, this might help:

https://wiki.shibboleth.net/confluence/display/SP3/WindowsBuild

And feed back on those pages welcome...

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
The BuildPath.props is missing from the zip, do you have an example
somewhere so I don't have to figure out what's supposed to be in it?

--

Thanks,

Dan McLaughlin
Technology Consortium, LLC
[hidden email]
mobile: 512.633.8086
http://www.tech-consortium.com

NOTICE: This e-mail message and all attachments transmitted with it
are for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is strictly prohibited. The contents of
this e-mail are confidential and may be subject to work product
privileges. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message.

On Fri, Sep 20, 2019 at 4:04 AM Rod Widdowson <[hidden email]> wrote:

>
> > Sounds good, in the meantime, I'm trying to see if I can build it
> > myself from the zip file I downloaded.  I've never built it before, so
> > I'm assuming that's enough.
>
> If you are not aware of it, this might help:
>
> https://wiki.shibboleth.net/confluence/display/SP3/WindowsBuild
>
> And feed back on those pages welcome...
>
> --
> To unsubscribe from this list send an email to [hidden email]
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP Crashing

Rod Widdowson
> The BuildPath.props is missing from the zip, do you have an example
> somewhere so I don't have to figure out what's supposed to be in it?

(Replied offline)

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
In reply to this post by Dan McLaughlin
There's a tree with all the binaries zipped up in downloads/misc/VC15 on the server. I don't know how usable it would be, but assuming an install so all the schemas were already in the right spot, using these in the path should work as long as the config variable is set to locate the configuration.

But this doesn't include all the program database files and I don't have any convenient way to include those, so apart from probably crashing in perhaps more frequent or predictable ways if there's a bug it may not reveal much.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
Thanks!  Luckily I stumbled across those on accident before I started.
But I could see myself trying to put it all together manually had I
not.  Something like
https://github.com/centic9/subversion-ppa/blob/master/get-deps.sh in
the build that would automatically get all the dependencies would be
nice, or better yet, use a cross platform dependency management tool
similar to maven.

Regardless, It would be helpful to refer to those files in the doc for
sure--and add an example BuildPath.props.  I'd also recommend you
rearrange the doc so that important information like the sentence "The
build takes configuration from environmental variable set up in the
file cpp-msbuild/dependencies/config.bat" is placed before the table,
not after, because in my experience people sometimes miss information
when it's after.

BTW... the docs keep referring to a cpp-msbuild directory a config.bat
and a versions.props file that don't exist anywhere in the shibboleth
zip file.  Is there another file I need to download that contains the
cpp-msbuild artifacts?

--

Thanks,

Dan McLaughlin
Technology Consortium, LLC
[hidden email]
mobile: 512.633.8086
http://www.tech-consortium.com

NOTICE: This e-mail message and all attachments transmitted with it
are for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is strictly prohibited. The contents of
this e-mail are confidential and may be subject to work product
privileges. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message.
On Fri, Sep 20, 2019 at 11:00 AM Cantor, Scott <[hidden email]> wrote:

>
> There's a tree with all the binaries zipped up in downloads/misc/VC15 on the server. I don't know how usable it would be, but assuming an install so all the schemas were already in the right spot, using these in the path should work as long as the config variable is set to locate the configuration.
>
> But this doesn't include all the program database files and I don't have any convenient way to include those, so apart from probably crashing in perhaps more frequent or predictable ways if there's a bug it may not reveal much.
>
> -- Scott
>
>
> --
> To unsubscribe from this list send an email to [hidden email]
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
On 9/20/19, 12:34 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:

> Regardless, It would be helpful to refer to those files in the doc for
> sure--and add an example BuildPath.props.

We have never set a goal of facilitating anybody else building it on Windows, so those are purely internal docs for Rod and myself that I asked him to put together so I could get things working when he changed the build.

> BTW... the docs keep referring to a cpp-msbuild directory a config.bat
> and a versions.props file that don't exist anywhere in the shibboleth
> zip file.  Is there another file I need to download that contains the
> cpp-msbuild artifacts?

Git, it's a separate project [1]. There are no zip files unless you mean the source archives, and those are essentially for Unix, not Windows (the zip is just built by default by make dist). There is no "source distribution" that stands alone for building on Windows. You literally can't build this without the cpp-msbuild pieces now, at least not in any way that would be tolerable.

-- Scott

[1] https://git.shibboleth.net/view/?p=cpp-msbuild.git;a=summary 


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP Crashing

Rod Widdowson
It occurred to me overnight that before you go too far down the build world (which is very much targeted at "Scott or Rod"), you
might get something from a remote debug session.

Assuming you have Visual Studio on a different machine:

* Get the Shib symbols onto the machine with visual studio (it will also need to be connected to the internet to resolve the MS
symbols)
* Get the remote debugging tools installed[1] and started (you'll need to run it elevated) on the machine with shibd.
* On the machine with the symbols and visual studio attach to the shibd.exe on the target machine (Debug->Attach to
process->ConnectionTarget == machine with shibd on it).
* Break in and make sure that all the symbols are resolving (this will take a while as VS downloads the symbols for all the DLL's
that shibd links again)
* let it run and when shibd crashes it should break into the debugger and you might get a clean stack of what the problem is.

However to set expectations the "new improved heap" that has, I think, been in VS since 15 is not (IMO) very helpful on the
diagnosis front and the classic problem with a heap corruption is that it happen very often after the event.

[1] https://visualstudio.microsoft.com/downloads/?q=remote


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
The problem I have is I don't have access to the symbols for the 3.0.4 SP release, so I'm having to build my own to get the symbols.   Unless, you know of somewhere I can find the debug symbols.

--

Thanks,

Dan McLaughlin
Technology Consortium, LLC
mobile: 512.633.8086

NOTICE: This e-mail message and all attachments transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is strictly prohibited. The contents of this e-mail are confidential and may be subject to work product privileges. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

Need to schedule a meeting??? http://www.tungle.me/DanMcLaughlin


On Sat, Sep 21, 2019 at 4:08 AM Rod Widdowson <[hidden email]> wrote:
It occurred to me overnight that before you go too far down the build world (which is very much targeted at "Scott or Rod"), you
might get something from a remote debug session.

Assuming you have Visual Studio on a different machine:

* Get the Shib symbols onto the machine with visual studio (it will also need to be connected to the internet to resolve the MS
symbols)
* Get the remote debugging tools installed[1] and started (you'll need to run it elevated) on the machine with shibd.
* On the machine with the symbols and visual studio attach to the shibd.exe on the target machine (Debug->Attach to
process->ConnectionTarget == machine with shibd on it).
* Break in and make sure that all the symbols are resolving (this will take a while as VS downloads the symbols for all the DLL's
that shibd links again)
* let it run and when shibd crashes it should break into the debugger and you might get a clean stack of what the problem is.

However to set expectations the "new improved heap" that has, I think, been in VS since 15 is not (IMO) very helpful on the
diagnosis front and the classic problem with a heap corruption is that it happen very often after the event.

[1] https://visualstudio.microsoft.com/downloads/?q=remote


--
To unsubscribe from this list send an email to [hidden email]

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Cantor, Scott E.
On 9/23/19, 3:29 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:

> The problem I have is I don't have access to the symbols for the 3.0.4 SP release, so I'm having to build my own to get > the symbols.   Unless, you know of somewhere I can find the debug symbols.

I don't know which file(s) anymore would contain the missing parts (I don't know that I ever knew, which is partly why we stopped including the debug build, it wasn't doing any good anyway). The files, whatever they are, for the dependencies are in the uploaded build trees. I can just zip up the other three "mine" projects tonight and upload those, and whatever's missing should be in there somewhere since that's all I have.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP Crashing

Dan McLaughlin
Sounds good, thanks!  BTW...I tried switching to the 32bit shib.exe and the same issue occurs. Just wanted to rule out something that might have been x64 specific. I see you uploaded a cpp-sp.zip, is that what I should grab?

--

Thanks,

Dan McLaughlin
Technology Consortium, LLC
mobile: 512.633.8086

NOTICE: This e-mail message and all attachments transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is strictly prohibited. The contents of this e-mail are confidential and may be subject to work product privileges. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

Need to schedule a meeting??? http://www.tungle.me/DanMcLaughlin


On Mon, Sep 23, 2019 at 2:36 PM Cantor, Scott <[hidden email]> wrote:
On 9/23/19, 3:29 PM, "dev on behalf of Dan McLaughlin" <[hidden email] on behalf of [hidden email]> wrote:

> The problem I have is I don't have access to the symbols for the 3.0.4 SP release, so I'm having to build my own to get > the symbols.   Unless, you know of somewhere I can find the debug symbols.

I don't know which file(s) anymore would contain the missing parts (I don't know that I ever knew, which is partly why we stopped including the debug build, it wasn't doing any good anyway). The files, whatever they are, for the dependencies are in the uploaded build trees. I can just zip up the other three "mine" projects tonight and upload those, and whatever's missing should be in there somewhere since that's all I have.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP Crashing

Cantor, Scott E.
> Sounds good, thanks!  BTW...I tried switching to the 32bit shib.exe and the
> same issue occurs. Just wanted to rule out something that might have been x64
> specific. I see you uploaded a cpp-sp.zip, is that what I should grab?

I just finished all three uploads and checksummed them for verification, they should match what's on my laptop. All three projects are uploaded so one way or the other all the symbols and debug versions are there.

If they're all unpacked to the same directory along with all the dependencies, then it would match what I have here when I debug it.

The shibboleth-sp zip is still there also, but that's strictly the copied DLLs and EXEs merged together for ease of copying into a system.

I have other things to do right now but the next chance I have to look at it I'll try to get my jmeter scripts working again and do the normal load testing I generally do for stressing it out. I did run them a year or so back when I released 3.0.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP Crashing

Rod Widdowson
In reply to this post by Dan McLaughlin
> The problem I have is I don't have access to the symbols for the 3.0.4 SP release

We might look to putting the pdb's into the msi (as an optional add-on) this is what the OpenAFS installer does, but I've never (before) been convinced of the utility or making the install significantly larger.  I suppose we could bundle them into a different installer too.  What I would strongly suggest against would be a symbol server (simpler in some ways, but yet more surface to protect and something else to go wrong)/

I'll also observe, as someone who spends much of his life working with symbols, that just getting a stack can occasionally be educational.

/Rod

--
To unsubscribe from this list send an email to [hidden email]
12