Tracking down intermittent stale request errors from the IdP

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Tracking down intermittent stale request errors from the IdP

Wessel, Keith William
All,

Hoping someone can point me in the right direction. We have a technical user here on campus who has been getting intermittent stale request messages from the IdP. He'll sign into a service behind Shib then, when he signs into another, he'll be prompted to re-enter his password even though forced reauth wasn't requested and he still has a valid IdP session. After clicking the login button on that page, he'll get a stale request message in his browser instead of being prompted for 2FA.

When this happens, he can go back to the same starting point for logging into the service several times in a row and get the same results until, after a few tries, it randomly works. He can even try in a private browsing window and get the same results while it's still happening.

So far, this has all been in Chrome and hasn't been happening to other customers, or at least they're not reporting it. But he's been putting up with it every couple days for the last couple months. It happens to a variety of SPs, but he's never had it happen when logging into AWS (which he does frequently) which uses IDP-initiated SSO.

He's tried the obvious clearing his cache and disabling browser extensions.

All that the IdP reports in the log is a no such flow exception.

I suspect his browser is doing some funny caching that it shouldn't be. I'm tempted to have him re-install Chrome or spend a few days in another browser. Before I do, wondered if anyone had suggestions of other things I might try to help troubleshoot this.

Thanks,
Keith

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Tracking down intermittent stale request errors from the IdP

Andrew Morgan
On Fri, 1 Jun 2018, Wessel, Keith wrote:

> All,
>
> Hoping someone can point me in the right direction. We have a technical
> user here on campus who has been getting intermittent stale request
> messages from the IdP. He'll sign into a service behind Shib then, when
> he signs into another, he'll be prompted to re-enter his password even
> though forced reauth wasn't requested and he still has a valid IdP
> session. After clicking the login button on that page, he'll get a stale
> request message in his browser instead of being prompted for 2FA.
>
> When this happens, he can go back to the same starting point for logging
> into the service several times in a row and get the same results until,
> after a few tries, it randomly works. He can even try in a private
> browsing window and get the same results while it's still happening.
>
> So far, this has all been in Chrome and hasn't been happening to other
> customers, or at least they're not reporting it. But he's been putting
> up with it every couple days for the last couple months. It happens to a
> variety of SPs, but he's never had it happen when logging into AWS
> (which he does frequently) which uses IDP-initiated SSO.
>
> He's tried the obvious clearing his cache and disabling browser
> extensions.
>
> All that the IdP reports in the log is a no such flow exception.
>
> I suspect his browser is doing some funny caching that it shouldn't be.
> I'm tempted to have him re-install Chrome or spend a few days in another
> browser. Before I do, wondered if anyone had suggestions of other things
> I might try to help troubleshoot this.

Keith,

Is your IDP load balanced with multiple nodes?  This sounds like the
JSESSION is being lost, which can happen if his requests go to different
(un-replicated) nodes as part of the login process.

We ran into this problem when we moved some nodes into AWS behind the
Network Load Balancer.  The NLB does not support sticky sessions, so the
JSESSION and web flow would be lost when a client switched nodes between
the steps of the login flow.

Thanks,
  Andy
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Tracking down intermittent stale request errors from the IdP

Cantor, Scott E.
> Is your IDP load balanced with multiple nodes?  This sounds like the JSESSION is
> being lost, which can happen if his requests go to different
> (un-replicated) nodes as part of the login process.

I always forget to ask that.

That's far and away the most likely reason. When the client behavior doesn't make sense (e.g. the private window thing), that almost always means the server's involved somehow.

-- Scott

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Tracking down intermittent stale request errors from the IdP

Wessel, Keith William
In reply to this post by Andrew Morgan
Interesting possibility. You're suggesting he's switching nodes half way through the login process, right?

It is load balanced, but with session stickiness enabled and a five-minute sticky period. And during the time that this happened today, there were no hits from his IP address in the web server log. His last hit on the other node was at 12:09. He successfully logged into a service at 12:30 then got the error a minute later at 12:31 on the same node.

Keith


-----Original Message-----
From: users <[hidden email]> On Behalf Of Andrew Morgan
Sent: Friday, June 1, 2018 3:47 PM
To: Shib Users <[hidden email]>
Subject: Re: Tracking down intermittent stale request errors from the IdP

On Fri, 1 Jun 2018, Wessel, Keith wrote:

> All,
>
> Hoping someone can point me in the right direction. We have a technical
> user here on campus who has been getting intermittent stale request
> messages from the IdP. He'll sign into a service behind Shib then, when
> he signs into another, he'll be prompted to re-enter his password even
> though forced reauth wasn't requested and he still has a valid IdP
> session. After clicking the login button on that page, he'll get a stale
> request message in his browser instead of being prompted for 2FA.
>
> When this happens, he can go back to the same starting point for logging
> into the service several times in a row and get the same results until,
> after a few tries, it randomly works. He can even try in a private
> browsing window and get the same results while it's still happening.
>
> So far, this has all been in Chrome and hasn't been happening to other
> customers, or at least they're not reporting it. But he's been putting
> up with it every couple days for the last couple months. It happens to a
> variety of SPs, but he's never had it happen when logging into AWS
> (which he does frequently) which uses IDP-initiated SSO.
>
> He's tried the obvious clearing his cache and disabling browser
> extensions.
>
> All that the IdP reports in the log is a no such flow exception.
>
> I suspect his browser is doing some funny caching that it shouldn't be.
> I'm tempted to have him re-install Chrome or spend a few days in another
> browser. Before I do, wondered if anyone had suggestions of other things
> I might try to help troubleshoot this.

Keith,

Is your IDP load balanced with multiple nodes?  This sounds like the
JSESSION is being lost, which can happen if his requests go to different
(un-replicated) nodes as part of the login process.

We ran into this problem when we moved some nodes into AWS behind the
Network Load Balancer.  The NLB does not support sticky sessions, so the
JSESSION and web flow would be lost when a client switched nodes between
the steps of the login flow.

Thanks,
  Andy
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Tracking down intermittent stale request errors from the IdP

Jim Fox-2
This was our problem and solution, but might not be yours.
Session stickiness requires a mechanism.  We started with client IP affinity.  Turns out some of our users are behind routers that bounce between two or more addresses (I have no idea why).  We switched to cookie affinity.  Other users got the 'no flow' errors. My guess is that some of our users don't keep their clocks very accurate.  If they are ahead by more than our cookie lifetime the session cookies are discarded.  We finally adopted a cookie-based session with a fallback IP session (the F5 can do this).  That mostly ended our 'no flow' incidents.
Jim

________________________________________
From: users <[hidden email]> on behalf of Wessel, Keith <[hidden email]>
Sent: Friday, June 1, 2018 2:38:20 PM
To: Shib Users
Subject: RE: Tracking down intermittent stale request errors from the IdP

Interesting possibility. You're suggesting he's switching nodes half way through the login process, right?

It is load balanced, but with session stickiness enabled and a five-minute sticky period. And during the time that this happened today, there were no hits from his IP address in the web server log. His last hit on the other node was at 12:09. He successfully logged into a service at 12:30 then got the error a minute later at 12:31 on the same node.

Keith


-----Original Message-----
From: users <[hidden email]> On Behalf Of Andrew Morgan
Sent: Friday, June 1, 2018 3:47 PM
To: Shib Users <[hidden email]>
Subject: Re: Tracking down intermittent stale request errors from the IdP

On Fri, 1 Jun 2018, Wessel, Keith wrote:

> All,
>
> Hoping someone can point me in the right direction. We have a technical
> user here on campus who has been getting intermittent stale request
> messages from the IdP. He'll sign into a service behind Shib then, when
> he signs into another, he'll be prompted to re-enter his password even
> though forced reauth wasn't requested and he still has a valid IdP
> session. After clicking the login button on that page, he'll get a stale
> request message in his browser instead of being prompted for 2FA.
>
> When this happens, he can go back to the same starting point for logging
> into the service several times in a row and get the same results until,
> after a few tries, it randomly works. He can even try in a private
> browsing window and get the same results while it's still happening.
>
> So far, this has all been in Chrome and hasn't been happening to other
> customers, or at least they're not reporting it. But he's been putting
> up with it every couple days for the last couple months. It happens to a
> variety of SPs, but he's never had it happen when logging into AWS
> (which he does frequently) which uses IDP-initiated SSO.
>
> He's tried the obvious clearing his cache and disabling browser
> extensions.
>
> All that the IdP reports in the log is a no such flow exception.
>
> I suspect his browser is doing some funny caching that it shouldn't be.
> I'm tempted to have him re-install Chrome or spend a few days in another
> browser. Before I do, wondered if anyone had suggestions of other things
> I might try to help troubleshoot this.

Keith,

Is your IDP load balanced with multiple nodes?  This sounds like the
JSESSION is being lost, which can happen if his requests go to different
(un-replicated) nodes as part of the login process.

We ran into this problem when we moved some nodes into AWS behind the
Network Load Balancer.  The NLB does not support sticky sessions, so the
JSESSION and web flow would be lost when a client switched nodes between
the steps of the login flow.

Thanks,
        Andy
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Tracking down intermittent stale request errors from the IdP

Wessel, Keith William
Thaks, Jim. Our Brocade SLBs handle session stickiness with an internal cache, not cookies. So, it shouldn't be clock skew or cookie lifetime. It's unlikely that he's changing IPs mid-authentication, but I won't rule it out. He was on campus wireless with the IP he gave me yesterday, and all of his connections seemed to be coming from the same IP.

Perhaps it's time to have him either re-install Chrome or spend a few days in a different browser to at least rule out browser funkiness, though that seems unlikely.

Keith


-----Original Message-----
From: users <[hidden email]> On Behalf Of Jim Fox
Sent: Friday, June 1, 2018 11:09 PM
To: Shib Users <[hidden email]>
Subject: Re: Tracking down intermittent stale request errors from the IdP

This was our problem and solution, but might not be yours.
Session stickiness requires a mechanism.  We started with client IP affinity.  Turns out some of our users are behind routers that bounce between two or more addresses (I have no idea why).  We switched to cookie affinity.  Other users got the 'no flow' errors. My guess is that some of our users don't keep their clocks very accurate.  If they are ahead by more than our cookie lifetime the session cookies are discarded.  We finally adopted a cookie-based session with a fallback IP session (the F5 can do this).  That mostly ended our 'no flow' incidents.
Jim

________________________________________
From: users <[hidden email]> on behalf of Wessel, Keith <[hidden email]>
Sent: Friday, June 1, 2018 2:38:20 PM
To: Shib Users
Subject: RE: Tracking down intermittent stale request errors from the IdP

Interesting possibility. You're suggesting he's switching nodes half way through the login process, right?

It is load balanced, but with session stickiness enabled and a five-minute sticky period. And during the time that this happened today, there were no hits from his IP address in the web server log. His last hit on the other node was at 12:09. He successfully logged into a service at 12:30 then got the error a minute later at 12:31 on the same node.

Keith


-----Original Message-----
From: users <[hidden email]> On Behalf Of Andrew Morgan
Sent: Friday, June 1, 2018 3:47 PM
To: Shib Users <[hidden email]>
Subject: Re: Tracking down intermittent stale request errors from the IdP

On Fri, 1 Jun 2018, Wessel, Keith wrote:

> All,
>
> Hoping someone can point me in the right direction. We have a technical
> user here on campus who has been getting intermittent stale request
> messages from the IdP. He'll sign into a service behind Shib then, when
> he signs into another, he'll be prompted to re-enter his password even
> though forced reauth wasn't requested and he still has a valid IdP
> session. After clicking the login button on that page, he'll get a stale
> request message in his browser instead of being prompted for 2FA.
>
> When this happens, he can go back to the same starting point for logging
> into the service several times in a row and get the same results until,
> after a few tries, it randomly works. He can even try in a private
> browsing window and get the same results while it's still happening.
>
> So far, this has all been in Chrome and hasn't been happening to other
> customers, or at least they're not reporting it. But he's been putting
> up with it every couple days for the last couple months. It happens to a
> variety of SPs, but he's never had it happen when logging into AWS
> (which he does frequently) which uses IDP-initiated SSO.
>
> He's tried the obvious clearing his cache and disabling browser
> extensions.
>
> All that the IdP reports in the log is a no such flow exception.
>
> I suspect his browser is doing some funny caching that it shouldn't be.
> I'm tempted to have him re-install Chrome or spend a few days in another
> browser. Before I do, wondered if anyone had suggestions of other things
> I might try to help troubleshoot this.

Keith,

Is your IDP load balanced with multiple nodes?  This sounds like the
JSESSION is being lost, which can happen if his requests go to different
(un-replicated) nodes as part of the login process.

We ran into this problem when we moved some nodes into AWS behind the
Network Load Balancer.  The NLB does not support sticky sessions, so the
JSESSION and web flow would be lost when a client switched nodes between
the steps of the login flow.

Thanks,
        Andy
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]