Issue with _shibsession_ in Load Balanced Setup

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue with _shibsession_ in Load Balanced Setup

Special Monkey
I have a 2 node load balanced phpMyAdmin setup. The nodes are behind an AWS ALB.

The cloud team mentioned this with regard to the ALB:

1 it utilizes cookies, so as long as clients browser settings are OK, the session will remain with a single node
2 the stickiness duration is currently set to 10 minutes, which means as long as requests are being sent between the client and lb within that time frame, the session will remain sticky so to speak
but if they walk away, 10 minutes elapses, then they start interacting with the app again, it's possible they get bounced to the other node

I notice the AWSALB cookie is constantly changing, I understand this is normal and is how stickiness works with the AWS ALB.

I also see the _shibsession_xxx session cookie.

Every so often, the app (phpMyAdmin) complains that it has lost its connection with the following error displayed in the web UI:

Error in processing request
Error text: error (rejected)
It seems that the connection to server has been lost. Please check your network connectivity and server status.


This happens after some idle time, and then clicking on, for example, a table, to see its contents (or seemingly any link within the app's web UI).  

The app is setup with protected paths in the Apache's shibd.conf file, for example:

<Location /secure/bas>
  AuthType shibboleth
  ShibRequestSetting requireSession true
<RequireAll>
  Require shib-attr isMemberOf cn=bas,ou=groups,ou=db,ou=app,ou=foo,ou=Groups,o=foo.com,o=foo
  Require shib-attr isMemberOf cn=everyone,ou=groups,ou=db,ou=app,ou=foo,ou=Groups,o=foo.com,o=foo
</RequireAll>
</Location>

each of which (i.e., /secure/bas, etc.) are phpMyAdmin installs with auth_type = config setups -- where user & password are hard coded into the config.inc.php file, since auth is handled by Shibboleth.

phpMyAdmin connects via TLS to a MariaDB cluster. The connect details to MariaDB are also within each config.inc.php.

The following is in shibboleth2.xml:

<Sessions lifetime="28800" timeout="3600" relayState="ss:mem"
          checkAddress="false" consistentAddress="false" handlerSSL="true" cookieProps="https">

I set session.gc_maxlifetime = 14400 in php.ini. The server is using Red Hat:

- Apache 2.4.6
- Shibd 3.04
- PHP 7.3

Initially, one issue was resolved by setting consistentAddress="false" as above ... since the ALB actually has 2 source IPs at any time, and when they changed, it seemed to correspond to the error occurring.  

consistentAddress="false" seemed to resolve -- but the issue still cropped up, albeit less frequently.

I thought the issue might be related to the app not sharing sessions, so I tried configuring PHP to use Redis to share sessions between the 2 nodes, however, the issue still happens.

Deleting the phpMyAdmin session cookie does not seem to cause the problem.

Deleting the _shibsession_xxx cookie, DOES produce the error.

Some things I notice when the error happens,

- the link I click responds with a 302 to the IdP
- the node that responded with the 302 is different from the node the browser had been on
- the _shibsession_xxx session cookie that had been present disappears from local storage, and the app error:

Error in processing request
Error text: error (rejected)
It seems that the connection to server has been lost. Please check your network connectivity and server status.


is presented in the app's web UI.

I see in the 302 provisional headers a _shibsession_xxx and a _shibsealed_xxx cookie -- the names for both are the same. I can't see the values.

Reloading the page fixes the error, and now the browser is sticking to the alternate node. The _shibsession_xxx cookie is back and has a new value on reloading. Everything is fine until it switches again.

Thanks for any info on how I might debug.

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Issue with _shibsession_ in Load Balanced Setup

Special Monkey
I spoke with our IdP admin who suggested making the ALB timeout higher than the <Sessions lifetime="28800" (in shibboleth2.xml) or above 8 hours -- it had been 10 mins.

This seems to be a step in the right direction. I haven't seen the phpMyAdmin error again since upping the ALB timeout to 9 hours.

The long ALB timeout is a bit of a Band-Aid solution since it essentially prevents the node from switching during a typical browser session, which the app cannot seem to handle gracefully, i.e. without eventually presenting the reported error -- even with shared PHP sessions.

The app is relying on the Shibboleth session and not the PHP session -- seemingly proved when deleting the phpMyAdmin session cookie does not cause the error but deleting the Shibboleth session cookie does.

A Band-Aid is arguably better than no Band-Aid. I think this one is limiting the load balancing by remaining so sticky, but is utilizing some aspects of load balancing by directing clients to different nodes, and if a node goes down, to the node that is hopefully still up, which I think would cause the same app error, though much less frequently (e.g., only when a node the client is on becomes unavailable or after 9 hours?).

I'm curious what the recommended approach to this issue is, assuming something similar happens to other load balanced Shibboleth protected apps?
Since the app is able to regain a session on a reload of the page, from the different node's SP setup, I wonder if the app (or some SP config change) could make an accommodation, or check for a new session, before producing the error, and requiring a reload of the page to continue.

The admin also referred me to the following:




On Thu, Nov 14, 2019 at 1:50 PM Special Monkey <[hidden email]> wrote:
I have a 2 node load balanced phpMyAdmin setup. The nodes are behind an AWS ALB.

The cloud team mentioned this with regard to the ALB:

1 it utilizes cookies, so as long as clients browser settings are OK, the session will remain with a single node
2 the stickiness duration is currently set to 10 minutes, which means as long as requests are being sent between the client and lb within that time frame, the session will remain sticky so to speak
but if they walk away, 10 minutes elapses, then they start interacting with the app again, it's possible they get bounced to the other node

I notice the AWSALB cookie is constantly changing, I understand this is normal and is how stickiness works with the AWS ALB.

I also see the _shibsession_xxx session cookie.

Every so often, the app (phpMyAdmin) complains that it has lost its connection with the following error displayed in the web UI:

Error in processing request
Error text: error (rejected)
It seems that the connection to server has been lost. Please check your network connectivity and server status.


This happens after some idle time, and then clicking on, for example, a table, to see its contents (or seemingly any link within the app's web UI).  

The app is setup with protected paths in the Apache's shibd.conf file, for example:

<Location /secure/bas>
  AuthType shibboleth
  ShibRequestSetting requireSession true
<RequireAll>
  Require shib-attr isMemberOf cn=bas,ou=groups,ou=db,ou=app,ou=foo,ou=Groups,o=foo.com,o=foo
  Require shib-attr isMemberOf cn=everyone,ou=groups,ou=db,ou=app,ou=foo,ou=Groups,o=foo.com,o=foo
</RequireAll>
</Location>

each of which (i.e., /secure/bas, etc.) are phpMyAdmin installs with auth_type = config setups -- where user & password are hard coded into the config.inc.php file, since auth is handled by Shibboleth.

phpMyAdmin connects via TLS to a MariaDB cluster. The connect details to MariaDB are also within each config.inc.php.

The following is in shibboleth2.xml:

<Sessions lifetime="28800" timeout="3600" relayState="ss:mem"
          checkAddress="false" consistentAddress="false" handlerSSL="true" cookieProps="https">

I set session.gc_maxlifetime = 14400 in php.ini. The server is using Red Hat:

- Apache 2.4.6
- Shibd 3.04
- PHP 7.3

Initially, one issue was resolved by setting consistentAddress="false" as above ... since the ALB actually has 2 source IPs at any time, and when they changed, it seemed to correspond to the error occurring.  

consistentAddress="false" seemed to resolve -- but the issue still cropped up, albeit less frequently.

I thought the issue might be related to the app not sharing sessions, so I tried configuring PHP to use Redis to share sessions between the 2 nodes, however, the issue still happens.

Deleting the phpMyAdmin session cookie does not seem to cause the problem.

Deleting the _shibsession_xxx cookie, DOES produce the error.

Some things I notice when the error happens,

- the link I click responds with a 302 to the IdP
- the node that responded with the 302 is different from the node the browser had been on
- the _shibsession_xxx session cookie that had been present disappears from local storage, and the app error:

Error in processing request
Error text: error (rejected)
It seems that the connection to server has been lost. Please check your network connectivity and server status.


is presented in the app's web UI.

I see in the 302 provisional headers a _shibsession_xxx and a _shibsealed_xxx cookie -- the names for both are the same. I can't see the values.

Reloading the page fixes the error, and now the browser is sticking to the alternate node. The _shibsession_xxx cookie is back and has a new value on reloading. Everything is fine until it switches again.

Thanks for any info on how I might debug.

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Issue with _shibsession_ in Load Balanced Setup

Cantor, Scott E.
On 11/15/19, 1:11 PM, "users on behalf of Special Monkey" <[hidden email] on behalf of [hidden email]> wrote:

> The admin also referred me to the following:

That's why the page was written, it answers everything you're asking. If you're looking for some kind of magic solution, there isn't one.
 
But the page doesn't have a reference to the cookie session recovery feature that allows movement between servers, I'll add it. It is however unstable at present until the next release.

-- Scott



--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]