Oddness running from a daemon.

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Oddness running from a daemon.

Rod Widdowson
I'm looking for guidance on debugging this one.

I'm seeing an issue when I run Jetty from a Daemon in Windows.  The symptom
is that during the authentication flow all goes OK until we despatch to
auth\Password:

2014-10-07 09:24:14,218 - DEBUG
[net.shibboleth.idp.authn.impl.SelectAuthenticationFlow:267] - Profile
Action SelectAuthenticationFlow: No usable active results available,
selecting an inactive flow
2014-10-07 09:24:14,218 - DEBUG
[net.shibboleth.idp.authn.impl.SelectAuthenticationFlow:309] - Profile
Action SelectAuthenticationFlow: Selecting inactive authentication flow
authn/Password
2014-10-07 09:24:14,234 - DEBUG
[net.shibboleth.ext.spring.error.ExtendedMappingExceptionResolver:134] -
Resolving exception from handler
[[FlowHandlerMapping.DefaultFlowHandler@183210e9]]:
org.springframework.webflow.execution.FlowExecutionException: Exception
thrown in state 'CallAuthenticationFlow' of flow 'authn'
2014-10-07 09:24:14,234 - DEBUG
[net.shibboleth.ext.spring.error.ExtendedMappingExceptionResolver:219] -
Resolving to default view 'error' for exception of type
[org.springframework.webflow.execution.FlowExecutionException]
2014-10-07 09:24:14,234 - ERROR
[org.springframework.webflow.execution.FlowExecutionException:76] -
org.springframework.webflow.execution.FlowExecutionException: Exception
thrown in state 'CallAuthenticationFlow' of flow 'authn'
        at
org.springframework.webflow.engine.impl.FlowExecutionImpl.wrap(FlowExecution
Impl.java:573)
Caused by:
org.springframework.webflow.definition.registry.FlowDefinitionConstructionEx
ception: An exception occurred constructing the flow 'authn/Password'
        at
org.springframework.webflow.engine.builder.DefaultFlowHolder.assembleFlow(De
faultFlowHolder.java:111)
Caused by: org.springframework.webflow.engine.builder.FlowBuilderException:
Unable to get the model for this flow
        at
org.springframework.webflow.engine.builder.model.FlowModelFlowBuilder.doInit
(FlowModelFlowBuilder.java:154)
Caused by:
org.springframework.webflow.engine.model.builder.FlowModelBuilderException:
Unable to find flow 'authn/conditions' to inherit from
        at
org.springframework.webflow.engine.model.builder.xml.XmlFlowModelBuilder.mer
geFlows(XmlFlowModelBuilder.java:635)
Caused by:
org.springframework.webflow.engine.model.registry.NoSuchFlowModelException:
No flow model 'authn/conditions' found
        at
org.springframework.webflow.engine.model.registry.FlowModelRegistryImpl.getF
lowModelHolder(FlowModelRegistryImpl.java:80)
2014-10-07 09:24:14,249 - DEBUG
[net.shibboleth.ext.spring.error.ExtendedMappingExceptionResolver:341] -
Exposing Exception as model attribute 'exception'


This works just fine from the command line, so it is something odd about the
fact that I'm running from a service.  I watched the process doing a lookup
in "authn\conditions\*" and now I'm currently floored so any suggestions
about what precisely is up with that transition would be welcome.  

An aside: while I was debugging this I did spot an oddity.  The daemon
wanted to load a file called:

C:\Program Files
(x86)\Shibboleth\Jetty\resources\%{idp\storage\ClientStorageService:org\open
saml\storage\impl\ServletRequestScopedStorageService}.class

I'm sure that has nothing to do with problem in hand, but I'll bring it to
your attention in case it rings any bells.

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Marvin Addison
> Caused by:
> org.springframework.webflow.engine.model.registry.NoSuchFlowModelException:
> No flow model 'authn/conditions' found

Reads like a resource not found, and that makes me wonder if it's a
matter of the process current working directory being different for a
service versus user-launched process. Assuming that Spring uses
relative paths for resource loading, a difference in working directory
could explain the problem. Why it happens there and not long before
remains a mystery.

M
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
In reply to this post by Rod Widdowson
On 10/7/14, 5:24 AM, "Rod Widdowson" <[hidden email]> wrote:
>
>This works just fine from the command line, so it is something odd about
>the
>fact that I'm running from a service.  I watched the process doing a
>lookup
>in "authn\conditions\*" and now I'm currently floored so any suggestions
>about what precisely is up with that transition would be welcome.

Well, my guess is we're seeing an issue with the pattern-based flow
mappings, which is all pretty wacky stuff.

When you say it's looking for "authn\conditions\*" what does that mean
exactly? Which layer was doing that? You mean it was walking that
directory?


>An aside: while I was debugging this I did spot an oddity.  The daemon
>wanted to load a file called:
>
>C:\Program Files
>(x86)\Shibboleth\Jetty\resources\%{idp\storage\ClientStorageService:org\op
>en
>saml\storage\impl\ServletRequestScopedStorageService}.class
>
>I'm sure that has nothing to do with problem in hand, but I'll bring it to
>your attention in case it rings any bells.

Unrelated probably, but it looks like maybe there's an issue in a property
somewhere, something not getting replaced. But I would think that would
blow up. I can't imagine Spring actually would look at the un-replaced
value and try to use it, so that's pretty odd.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
In reply to this post by Rod Widdowson
On 10/7/14, 5:24 AM, "Rod Widdowson" <[hidden email]> wrote:
>
>An aside: while I was debugging this I did spot an oddity.  The daemon
>wanted to load a file called:
>
>C:\Program Files
>(x86)\Shibboleth\Jetty\resources\%{idp\storage\ClientStorageService:org\op
>en
>saml\storage\impl\ServletRequestScopedStorageService}.class

Are you seeing it load anything else of that nature, or can't you tell?
There's only one spot that shows up, and it's formatted the same as other
property-based classes in the same file.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
In reply to this post by Rod Widdowson
On 10/7/14, 5:24 AM, "Rod Widdowson" <[hidden email]> wrote:
>
>This works just fine from the command line, so it is something odd about
>the
>fact that I'm running from a service.  I watched the process doing a
>lookup
>in "authn\conditions\*" and now I'm currently floored so any suggestions
>about what precisely is up with that transition would be welcome.

Refreshing my memory, it should be doing that kind of lookup during the
initialization of the flow repository, because the pattern-based element
tells it to walk the directory tree under the base-path to look for files
that match the pattern and then register the flow files under a flow ID it
computes.

So what I would do is debug into that process and see what it's actually
registering. It's clearly not registering authn/conditions, obviously. I
can track down the exact class to trace if you need me to, I don't recall
it offhand but I did find it before.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Tom Zeller-3
>>This works just fine from the command line, so it is something odd about
>>the
>>fact that I'm running from a service.  I watched the process doing a
>>lookup
>>in "authn\conditions\*" and now I'm currently floored so any suggestions
>>about what precisely is up with that transition would be welcome.
>
> Refreshing my memory, it should be doing that kind of lookup during the
> initialization of the flow repository, because the pattern-based element
> tells it to walk the directory tree under the base-path to look for files
> that match the pattern and then register the flow files under a flow ID it
> computes.
>
> So what I would do is debug into that process and see what it's actually
> registering. It's clearly not registering authn/conditions, obviously. I
> can track down the exact class to trace if you need me to, I don't recall
> it offhand but I did find it before.

If I should look at this, let me know. I have a custom build of SWF
that has additional logging when flow definitions are registered in
the repository. I would need to get my Windows VM up again, not a big
deal.
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
In reply to this post by Marvin Addison
Thanks all for your input...

Marvin:

>  wonder if it's a
> matter of the process current working directory being different for a
> service versus user-launched process.

Almost certainly.  Windows Daemons do not really have a working directory
(and Java helpfully provides no mechanism to set it...)

I thought I had tried running this from a wrong working directory (and
specify idp.home as a system property), but I may have fat fingered it.
That's next up.

Scott:
> When you say it's looking for "authn\conditions\*" what does that mean
> exactly? Which layer was doing that? You mean it was walking that
> directory?

Literally - watching with procmon (Win32 level) I see the process issue
QueryDirectory calls to recursively enumerate "flows\authn\*", then
"flows\authn\conditions\*", "flows\authn\conditions\account-locked\*" and so
on...

What I then went on to observe this morning (but ****ng outlook sent my mail
mentioning it back to me rather than to shib-dev) are these log entries
(early on - and with all spring logging turned on which is why I missed them
first time)

2014-10-07 10:26:21,069 - DEBUG
[org.springframework.webflow.definition.registry.FlowDefinitionRegistryImpl:
99] - Registering flow definition 'file [C:\Program Files
(x86)\Shibboleth\IdP\flows\access\attr-check\attr-check-flow.xml]' under id
'C:/Program Files (x86)/Shibboleth/IdP/flows/access/attr-check'
2014-10-07 10:26:21,069 - DEBUG
[org.springframework.webflow.definition.registry.FlowDefinitionRegistryImpl:
99] - Registering flow definition 'file [C:\Program Files
(x86)\Shibboleth\IdP\flows\authn\conditions\account-locked\account-locked-fl
ow.xml]' under id 'C:/Program Files
(x86)/Shibboleth/IdP/flows/authn/conditions/account-locked'
2014-10-07 10:26:21,069 - DEBUG
[org.springframework.webflow.definition.registry.FlowDefinitionRegistryImpl:
99] - Registering flow definition 'file [C:\Program Files
(x86)\Shibboleth\IdP\flows\authn\conditions\conditions-flow.xml]' under id
'C:/Program Files (x86)/Shibboleth/IdP/flows/authn/conditions'

Which figures from what Scott says:

> , it should be doing that kind of lookup during the initialization of the
flow repository

So we are registering, but we are registering with the Fully Qualified Path,
as ID.  I'll check out the code.

> So what I would do is debug into that process and see what it's actually
> registering. It's clearly not registering authn/conditions, obviously. I
> can track down the exact class to trace if you need me to, I don't recall
> it offhand but I did find it before.

Ugh.  Java debugging a remote process (this is on a VM so I don't brick my
dev machine).  I'll do some more checking with differing home directories,
and with the daemon running as a forground proccess and see if I can find a
relatively safe reproducer.

Thanks all.

R

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
On 10/7/14, 11:18 AM, "Rod Widdowson" <[hidden email]> wrote:
>
>So we are registering, but we are registering with the Fully Qualified
>Path,
>as ID.  I'll check out the code.

Yeah, that's seemingly what I had tried to fix way back when, so it
regressed.

>Ugh.  Java debugging a remote process (this is on a VM so I don't brick my
>dev machine).  I'll do some more checking with differing home directories,
>and with the daemon running as a forground proccess and see if I can find
>a
>relatively safe reproducer.

I suspect this will be reproducible under normal conditions on Windows
without requiring a daemon, just getting the configs into a directory and
relying on idp.home.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
> I suspect this will be reproducible under normal conditions on Windows
> without requiring a daemon, just getting the configs into a directory and
> relying on idp.home.

Confirmed.

I'll move this all over to my dev machine and debug it there.  I'll let you
know what I find.

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
On 10/8/14, 5:58 AM, "Rod Widdowson" <[hidden email]> wrote:

>> I suspect this will be reproducible under normal conditions on Windows
>> without requiring a daemon, just getting the configs into a directory
>>and
>> relying on idp.home.
>
>Confirmed.
>
>I'll move this all over to my dev machine and debug it there.  I'll let
>you
>know what I find.

I fixed this once before, so let me know if you need a hand. We'll
probably have to collectively decide on a fix anyway, because it probably
means we need more than just the single idp.home variable we were shooting
for.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
> I fixed this once before, so let me know if you need a hand.

It's being a bit painfull right now as I work out how to get eclipse to
auto-load the spring sources.  So I'm mostly struggling the with the
environment rather than the debug (not got there yet)

I'll welcome suggestion on how to short cut that - for instance, can I build
a war file which is empty of all the jar files and rely on the eclipse
classpath to collect that?

> We'll
> probably have to collectively decide on a fix anyway, because it probably
> means we need more than just the single idp.home variable we were shooting
> for.

What a pain, but we know the worst of that.

Rod

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
I should add that the command line reproducer is

1) Download Jetty to % JETTY_HOME%
2) Unload and config the IdP to % IDP_HOME %
3) Make sure you are *not* in %IDP_HOME%

Java -Didp.home=%IDP_HOME% -jar %JETTY_HOME%\start.jar
jetty.base=%IDP_HOME%\jetty-base

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Marvin Addison
> I should add that the command line reproducer is
>
> 1) Download Jetty to % JETTY_HOME%
> 2) Unload and config the IdP to % IDP_HOME %
> 3) Make sure you are *not* in %IDP_HOME%
>
> Java -Didp.home=%IDP_HOME% -jar %JETTY_HOME%\start.jar
> jetty.base=%IDP_HOME%\jetty-base

Just curious whether the conditions to reproduce are specific to
Windows. I believe we meet those requirements on our dev IdPv3 on
Linux (CentOS) and have had no resource loading issues afaik.

M
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
On 10/9/14, 9:14 AM, "Marvin Addison" <[hidden email]> wrote:
>
>Just curious whether the conditions to reproduce are specific to
>Windows. I believe we meet those requirements on our dev IdPv3 on
>Linux (CentOS) and have had no resource loading issues afaik.

It's very likely specific to Windows, it's likely an issue with path
normalization somewhere because of slash type or drive letters. The flow
base-path isn't being matched as a prefix of the actual flow files being
loaded.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Cantor, Scott E.
In reply to this post by Rod Widdowson
> I should add that the command line reproducer is
>
> 1) Download Jetty to % JETTY_HOME%
> 2) Unload and config the IdP to % IDP_HOME %
> 3) Make sure you are *not* in %IDP_HOME%
>
> Java -Didp.home=%IDP_HOME% -jar %JETTY_HOME%\start.jar
> jetty.base=%IDP_HOME%\jetty-base

The current jetty example really only works if you're running that java command from jetty-base. The paths to the credentials for SSL are not really relative to the jetty.base property but to the jetty-base *directory* itself provided the CWD is in fact jetty-base.

So you have to be in there to start Jetty up. But assuming that qualifies as "not in IDP_HOME", which it seems to (it's inside there, but one level down), I can't reproduce this.

I installed to C:\opt\shibboleth-idp, but set -Didp.home=C:/opt/shibboleth-idp

The process log shows it installing the flows to the proper IDs. I have more set up to do to run the IdP and get to the login page, and then I can verify if the special user flows run.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
> I installed to C:\opt\shibboleth-idp, but set
-Didp.home=C:/opt/shibboleth-idp
>
> The process log shows it installing the flows to the proper IDs. I have
more set
> up to do to run the IdP and get to the login page, and then I can verify
if the
> special user flows run.

I've just duplicated this (and I can get to the login page before things go
south, but that’s my LDAP setup).

I apologise if I have sent you on a wild goose chase,  I was sure that I had
tested this and something else bad happened (to do with our old friend :/
being taken as a URL separator), but I must have fixed that somewhere else
in the setup.

FYI:

> So you have to be in there to start Jetty up.

I actually find that specifying jetty.base="%IDP_HOME%\jetty-base" as a
parameter to the java program allows you to fix that.

Rod



--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
On 10/10/14, 3:32 AM, "Rod Widdowson" <[hidden email]> wrote:
>
>I've just duplicated this (and I can get to the login page before things
>go
>south, but that¹s my LDAP setup).

What have you duplicated? It's still not working for you?

Can you give me an example path to set idp.home to?

>>So you have to be in there to start Jetty up.
>
>I actually find that specifying jetty.base="%IDP_HOME%\jetty-base" as a
>parameter to the java program allows you to fix that.

That did not work for me, it couldn't locate the SSL files that are in
idp.ini, it just looked in the CWD's parent for creds.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
> That did not work for me, it couldn't locate the SSL files that are in
> idp.ini, it just looked in the CWD's parent for creds.

Ah yes, now you mention it, I did have that issue so I now dump the new
properties into a file which the installer merges into idp.ini (I also need
to potentially change the keystoretype to JCE if this is a V2 upgrade)

R

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Oddness running from a daemon.

Rod Widdowson
To wrap this up,

- My major problem was that I had idp.home specified with windows native
separators, not java/Unix native separators.  I'm wondering whether we need
to capture this bug in a FAQ area? I'm guessing its going to be a common one
- unless of course all windows users use the installer to install Jetty at
which stage they'll not see the problem.

- With some minor tinkering to my ldap-authn config I can now get logged in
and back successfully to the SP.

For the record, these are the properties changed in conf\idp.properties:

idp.entityID
idp.sealer.storePassword
idp.sealer.keyPassword
idp.scope

And these are the properties changed in jetty-base\start.d\idp.ini (for
Jetty)

jetty.host=0.0.0.0
jetty.https.port=443
jetty.backchannel.port=8443
jetty.backchannel.keystore.path=C:/Program Files
(x86)/Shibboleth/IdP/creds/idp-backchannel.p12
jetty.browser.keystore.path=C:/Program Files
(x86)/Shibboleth/IdP/creds/idp-userfacing.p12
jetty.backchannel.keystore.password
jetty.browser.keystore.password
jetty.backchannel.keystore.type=PKCS12
jetty.browser.keystore.type=PKCS12
jetty.war.path=C:/Program Files (x86)/Shibboleth/IdP/idp.war
jetty.jaas.path=C:/Program Files (x86)/Shibboleth/IdP/conf/authn/jaas.config

I'll make the changes to the MSI installer to ake sure that idp.home is
always set correctly (for jetty installs).  

Again, I'd like to apologise to Scott for sending on a wild goose chase with
this one.

Rod

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Oddness running from a daemon.

Cantor, Scott E.
On 10/11/14, 6:51 AM, "Rod Widdowson" <[hidden email]> wrote:

>- My major problem was that I had idp.home specified with windows native
>separators, not java/Unix native separators.  I'm wondering whether we
>need
>to capture this bug in a FAQ area? I'm guessing its going to be a common
>one
>- unless of course all windows users use the installer to install Jetty at
>which stage they'll not see the problem.

Yeah, certainly we document it clearly/loudly, and put comments in files,
etc., but my assumption is the majority of people will install to
/opt/shibboleth-idp or use the installer, and I believe that's what we
should encourage.

Anyway, while it's a minor hassle, it turns out to radically simplify
everything we built to stick with that convention.

>Again, I'd like to apologise to Scott for sending on a wild goose chase
>with
>this one.

I needed to get a Windows env back anyway, so it's now or later. Plus I
had to test the batch scripts (which you probably should test also to
check my work).

-- Scott

--
To unsubscribe from this list send an email to [hidden email]