Shib IdP - Metadata Download and Java 1.7.0_85

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Shib IdP - Metadata Download and Java 1.7.0_85

Wolfgang Pempe
Dear all,

recently, a java update 1.7.0_85 became available - at least for RHEL
and SL. After having installed this update, some of our users reported
on problems with the download of federation metadata
(FileBackedHTTPMetadataProvider), e.g.

2015-07-22 10:05:27.001 - ERROR
[org.opensaml.saml2.metadata.provider.HTTPMetadataProvider:273] -
[Timer-0:] - Error retrieving metadata from
https://www.aai.dfn.de/fileadmin/metadata/DFN-AAI-Basic-metadata.xml
javax.net.ssl.SSLPeerUnverifiedException: SSL peer failed hostname
validation for name: 194.95.248.235

This behaviour seems to go back to a security fix which is described at
http://www.oracle.com/technetwork/java/javase/7u85-relnotes-2587591.html
(end of the page).

Even though the workaround is obvious (-Djdk.tls.trustNameService=true),
I'm wondering whether this is an intended feature (well, should better
ask the java developers) which may presumably cause many other web
applications to 'break' - or do we have to deal with a Shib IdP-specific
problem?

Thanks,
Wolfgang
--
---------------------------------------------------------------------
Wolfgang Pempe                         Phone  : +49 711 63314-208
DFN-Verein, Geschäftsstelle Stuttgart  Fax    : +49 711 63314-133
Lindenspürstr.32                       E-Mail : [hidden email]
D-70176 Stuttgart                      WWW    : http://www.dfn.de
---------------------------------------------------------------------
--------------------- Deutsches Forschungsnetz ----------------------
--------- Germany's National Research and Education Network ---------
---------------------------------------------------------------------


--
To unsubscribe from this list send an email to [hidden email]

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
On 7/22/15, 8:14 AM, "users on behalf of Wolfgang Pempe" <[hidden email] on behalf of [hidden email]> wrote:



>This behaviour seems to go back to a security fix which is described at
>http://www.oracle.com/technetwork/java/javase/7u85-relnotes-2587591.html
>(end of the page).

I thought that bug had to do with an odd quirk that was allowing Java to validate connections to systems based on IP address by looking up the name and then matching that the certificate. Unless your connection URL included an IP address, that wouldn't apply, so one of us is confused.

Either way, as far as I knew, we're doing our own hostname checking anyway, unless you're running an older version (and even then I thought it was being done by another library, not Java's verifier, but I could be wrong on that).

Signed metadata with appropriate validUntil constraints can be hosted with or without TLS and with no hostname check.

>Even though the workaround is obvious (-Djdk.tls.trustNameService=true),

If that fixes this, then I'm totally lost.

>
>I'm wondering whether this is an intended feature (well, should better
>ask the java developers) which may presumably cause many other web
>applications to 'break' - or do we have to deal with a Shib IdP-specific
>problem?

I'm not sure I understand the question there, maybe you can rephrase.

Brent would have to address my confusion, and what code's really being used where, but unless this is a supported IdP version, that would obviously be step one in terms of providing a consistent answer.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Brent Putman


On 7/22/15 10:07 AM, Cantor, Scott wrote:
On 7/22/15, 8:14 AM, "users on behalf of Wolfgang Pempe" [hidden email] wrote:



This behaviour seems to go back to a security fix which is described at
http://www.oracle.com/technetwork/java/javase/7u85-relnotes-2587591.html
(end of the page).
I thought that bug had to do with an odd quirk that was allowing Java to validate connections to systems based on IP address by looking up the name and then matching that the certificate. Unless your connection URL included an IP address, that wouldn't apply, so one of us is confused.

That's what I thought too.  The OP's log seems to be showing the complete opposite case (URL w/ FQDN, failing validation against the IP address), and I'm not immediately seeing how they are connected.

For the original issue in v2 where someone noticed IP address metadata URLs mysteriously passing validation [1], I'm still not 100% sure why it was.  Only thing I can think of is:  we evaluate the cert against what is returned from SSLSession#getPeerHost().  If *that* is the place (or one of the places) where they were doing the reverse lookup, then that makes sense.  If not, then I still don't know how it was working there.

*******************

But on the changes in 7u85:  Researching this made me look into something that I had recently seen while poking around in JSSE.  This is new to me but: Apparently starting in Java 7, they introduced the capability for name checking to be done down in the TLS layer itself, calling it "endpoint identification" [2]:

"Endpoint verification: An endpoint identification algorithm can be specified to verify that a remote computer's host address matches its supplied certificate. Although this type of verification was previously performed for the HTTPS protocol (see HttpsURLConnection and HostnameVerifier), such verification can now be optionally performed at the TLS level."

I believe it's optional and not active by default.  You enable on the SSLContext and/or SSLSocket. (Don't know yet whether HttpClient factories or anything else we use is actually enabling this, but I doubt it). The default supported algorithms are "HTTPS" and "LDAPS" [3].

It's implemented (at least) in Oracle's X509ExtendedTrustManager [4] which I think you get by default with an SSLContext unless you otherwise specify one. I confirmed the behavior by looking at OpenJDK source code [5].

Soooo... My interpretation of the wording of the recent change:

" With this fix, JSSE endpoint identification does not perform reverse name lookup for IP addresses by default in JDK."

is that they are changing what this newer endpoint identification stuff does.  Although maybe they don't mean it quite that literally. Maybe it also extends to their legacy JDK HostnameVerifier impls (which we don't use).

So I am also not sure yet whether, where or how this change even affects our code (unless as mentioned it also affects the effective value returned from SSLSession#getPeerHost()).


Either way, as far as I knew, we're doing our own hostname checking anyway, unless you're running an older version (and even then I thought it was being done by another library, not Java's verifier, but I could be wrong on that).

That's correct.  I'm not seeing how this change affects our hostname verification process in either v2 or v3, given that we aren't using AFAIK any JDK code for that, nor does not-yet-commons-ssl nor HttpClient v4 (modulo SSLSession#getPeerHost() in v2. Not applicable in v3).  Based on the log, the OP seems to be on v2.

Basically if this new endpoint identification functionality *were* running, it would be a separate check from ours, I think. So it's possible that something might pass one check and fail the other, I guess. 




Even though the workaround is obvious (-Djdk.tls.trustNameService=true),
If that fixes this, then I'm totally lost.

Yeah, I wasn't clear whether this was confirmed to fix it, or just assumed.  If it does, then I also do not understand why.  Possibly the scope and nature of the change is bigger than what they have described.
 



[1] https://issues.shibboleth.net/jira/browse/JOWS-39
[2] http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements-7.html
[3] http://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html#jssenames
[4] https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509ExtendedTrustManager
[5] http://www.docjar.com/html/api/sun/security/ssl/X509TrustManagerImpl.java.html



--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
On 7/22/15, 6:39 PM, "users on behalf of Brent Putman" <[hidden email] on behalf of [hidden email]> wrote:


>
>For the original issue in v2 where someone noticed IP address metadata URLs mysteriously passing validation [1], I'm still not 100% sure why it was.  Only thing I can think of is:  we evaluate the cert against what is returned from SSLSession#getPeerHost().
> If *that* is the place (or one of the places) where they were doing the reverse lookup, then that makes sense.  If not, then I still don't know how it was working there.

That seems plausible. What a mess.

>That's correct.  I'm not seeing how this change affects our hostname verification process in either v2 or v3, given that we aren't using AFAIK any JDK code for that, nor does not-yet-commons-ssl nor HttpClient v4 (modulo SSLSession#getPeerHost() in v2. Not
> applicable in v3).  Based on the log, the OP seems to be on v2.

That confirms my understanding. I didn't think we were routing anything through that code.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Takeshi NISHIMURA
I think you should not rely on sslSession.getPeerHost() returning original hostnames. It seems like returning IP addresses on the latest versions of JDK.

This issue may be resolved if you pass the original hostname to verifyHostname() in [1].

Sincerely,
Takeshi

[1] http://svn.shibboleth.net/view/java-openws/branches/REL_1/src/main/java/org/opensaml/ws/soap/client/http/TLSProtocolSocketFactory.java?view=markup

On 2015/07/23 10:16, Cantor, Scott wrote:

> On 7/22/15, 6:39 PM, "users on behalf of Brent Putman" <[hidden email] on behalf of [hidden email]> wrote:
>
>
>>
>> For the original issue in v2 where someone noticed IP address metadata URLs mysteriously passing validation [1], I'm still not 100% sure why it was.  Only thing I can think of is:  we evaluate the cert against what is returned from SSLSession#getPeerHost().
>> If *that* is the place (or one of the places) where they were doing the reverse lookup, then that makes sense.  If not, then I still don't know how it was working there.
>
> That seems plausible. What a mess.
>
>> That's correct.  I'm not seeing how this change affects our hostname verification process in either v2 or v3, given that we aren't using AFAIK any JDK code for that, nor does not-yet-commons-ssl nor HttpClient v4 (modulo SSLSession#getPeerHost() in v2. Not
>> applicable in v3).  Based on the log, the OP seems to be on v2.
>
> That confirms my understanding. I didn't think we were routing anything through that code.
>
> -- Scott
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Brent Putman


On 7/23/15 12:26 AM, Takeshi NISHIMURA wrote:
I think you should not rely on sslSession.getPeerHost() returning original hostnames.

Well, hindsight is 20/20.  But I think I modeled that code on something "official" at the time, although I don't remember what.   And I'd note that that's exactly what they do in their own X509TrustMangerImpl [1] for the new endpoint identification stuff that I earlier mentioned:

if (identityAlg != null && identityAlg.length() != 0) {
    String hostname = session.getPeerHost();
    checkIdentity(hostname, chain[0], identityAlg);
}

So I thought this was the right way.  I do know however that the HttpClient v4 socket factories don't do that, they use the hostname originally passed.


It seems like returning IP addresses on the latest versions of JDK.


I really did not believe that they could do something so egregious.  So I had to test it (on "old" vs. "new" Java 8, since I don't have and can't currently get the last Java 7 with this change).  And in fact that is exactly what they have done.  By default SSLSession#getPeerHost() now returns the IP address, at least in conjunction with how the socket factory is used in HttpClient v3.  I think I read something earlier this evening that said the old quirky IP address behavior you experienced was only tripped if you used the socket factory methods that take an InetAddress as opposed to a String, or something.  So that's probably what the HttpClient is doing.

I'm flabbergasted.  I don't know if they intended this change or not.  It's certainly not what they were talking about in the release note, it's something different.  However, it's almost certainly related to the same set of changes because: Setting 'jdk.tls.trustNameService=true' causes getPeerHost() to return the hostname rather than the IP address.

So that is exactly the problem Wolfgang reported. So mystery solved, at least.

Setting jdk.tls.trustNameService=true' would be the workaround for the short term for v2, but looks like we'll have to issue a v2 patch update for this.  I don't think v3 would be affected, but we'll confirm.

As an aside, I don't see how their own X509TrustManagerImpl endpoint identification stuff (above) can work with this change, which is partly why I think it might have been an unintentional side effect of other changes. (Unless they do something even more egregious, like resolve the hostname(s) from the cert to IP addresses and match against those. Surely they can't be that daft...).


This issue may be resolved if you pass the original hostname to verifyHostname() in [1].


Yes, I'll confirm, but pretty sure that's what we should do.

Thanks for looking into this.


[1] http://www.docjar.com/html/api/sun/security/ssl/X509TrustManagerImpl.java.html


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Peter Schober
* Brent Putman <[hidden email]> [2015-07-23 08:20]:
> I'm flabbergasted.

Indeed.

Thanks everyone for checking, esp. Takeshi!

> As an aside, I don't see how their own X509TrustManagerImpl endpoint
> identification stuff (above) can work with this change, which is partly
> why I think it might have been an unintentional side effect of other
> changes. (Unless they do something even more egregious, like resolve
> the hostname(s) from the cert to IP addresses and match against those.
> Surely they can't be that daft...).

That would just break the other 99.9% of the web, presumably.

And people complain about Tomcat making breaking changes in patch
releases... ;)
-peter
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Peter Schober
* Peter Schober <[hidden email]> [2015-07-23 10:58]:
> > (Unless they do something even more egregious, like resolve
> > the hostname(s) from the cert to IP addresses and match against those.
> > Surely they can't be that daft...).
>
> That would just break the other 99.9% of the web, presumably.

I meant checking against the hostname returned from the PTR lookup.
-peter
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Wolfgang Pempe
In reply to this post by Peter Schober

Am 23.07.2015 um 10:59 schrieb Peter Schober:
> * Brent Putman <[hidden email]> [2015-07-23 08:20]:
>> I'm flabbergasted.
>
> Indeed.
>
> Thanks everyone for checking, esp. Takeshi!

+1

a possible federation-wide workaround (though not really politically
correct) would be to add the md server's IPv4 and v6 addresses as SaNs
to the cert. Perhaps that's what I'll do if the problem spreads...

Thanks again,
Wolfgang

>
>> As an aside, I don't see how their own X509TrustManagerImpl endpoint
>> identification stuff (above) can work with this change, which is partly
>> why I think it might have been an unintentional side effect of other
>> changes. (Unless they do something even more egregious, like resolve
>> the hostname(s) from the cert to IP addresses and match against those.
>> Surely they can't be that daft...).
>
> That would just break the other 99.9% of the web, presumably.
>
> And people complain about Tomcat making breaking changes in patch
> releases... ;)
> -peter
>
--
---------------------------------------------------------------------
Wolfgang Pempe                         Phone  : +49 711 63314-208
DFN-Verein, Geschäftsstelle Stuttgart  Fax    : +49 711 63314-133
Lindenspürstr.32                       E-Mail : [hidden email]
D-70176 Stuttgart                      WWW    : http://www.dfn.de
---------------------------------------------------------------------
--------------------- Deutsches Forschungsnetz ----------------------
--------- Germany's National Research and Education Network ---------
---------------------------------------------------------------------


--
To unsubscribe from this list send an email to [hidden email]

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
In reply to this post by Brent Putman
On 7/23/15, 2:20 AM, "users on behalf of Brent Putman" <[hidden email] on behalf of [hidden email]> wrote:
>
>I really did not believe that they could do something so egregious.  So I had to test it (on "old" vs. "new" Java 8, since I don't have and can't currently get the last Java 7 with this change).  And in fact that is exactly what they have done.

So this isn't in the original Java 8 but is now? What's the version cut off? Would be useful to document that.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Brent Putman


On 7/23/15 10:49 AM, Cantor, Scott wrote:
On 7/23/15, 2:20 AM, "users on behalf of Brent Putman" [hidden email] wrote:
I really did not believe that they could do something so egregious.  So I had to test it (on "old" vs. "new" Java 8, since I don't have and can't currently get the last Java 7 with this change).  And in fact that is exactly what they have done.
So this isn't in the original Java 8 but is now? What's the version cut off? Would be useful to document that.

Yes.  My understanding from the RedHat issue Takeshi originally posted [1] in JOWS-39 is that they "fixed" it in the July 2015 updates for all 3 versions: 6u101, 7u85, and 8u51

[1] https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2015-2625



--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
On 7/23/15, 1:04 PM, "users on behalf of Brent Putman" <[hidden email] on behalf of [hidden email]> wrote:
>
>Yes.  My understanding from the RedHat issue Takeshi originally posted [1] in JOWS-39 is that they "fixed" it in the July 2015 updates for all 3 versions: 6u101, 7u85, and 8u51

Thx.

I assume that setting disregardTlsCertificate also works around this if the TLS part is a formality?

I guess it's a good thing InCommon primarily documented http://

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Nate Klingenstein
I guess it's a good thing InCommon primarily documented http://

Better than that.  It doesn’t even host the files over https if you want.


—> 404

This does blow up a lot of services I’m running which are not tethered to specific IP addresses at all, so it’s going to be really fun for me personally.

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Brent Putman
In reply to this post by Brent Putman


On 7/23/15 2:20 AM, Brent Putman wrote:


On 7/23/15 12:26 AM, Takeshi NISHIMURA wrote:
I think you should not rely on sslSession.getPeerHost() returning original hostnames.

Well, hindsight is 20/20.  But I think I modeled that code on something "official" at the time, although I don't remember what.   And I'd note that that's exactly what they do in their own X509TrustMangerImpl [1] for the new endpoint identification stuff that I earlier mentioned:

In addition to OpenJDK X509TrustManagerImpl, googling around it seems that lots of JSSE examples on the internet use SSLSession#getPeerHost(), rightly or wrongly.

Closer to home, @Daniel:  I noticed that both vt-ldap and ldaptive are using this as well to get the hostname for verification:

vt-ldap: edu.vt.middleware.ldap.ssl.AbstractTLSSocketFactory

ldaptive: org.ldaptive.ssl.AbstractTLSSocketFactory, org.ldaptive.ssl.HostnameVerifyingListener

I have not actually tested them, so don't know if they trip this new behavior or not.


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Brent Putman
In reply to this post by Cantor, Scott E.


On 7/23/15 1:07 PM, Cantor, Scott wrote:
>
> I assume that setting disregardTlsCertificate also works around this if the TLS part is a formality?

Yes, it should work around, since that wires a null HostnameVerifier
and the verification is therefore completely skipped.

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
On 7/23/15, 1:18 PM, "users on behalf of Brent Putman" <[hidden email] on behalf of [hidden email]> wrote:
>
>On 7/23/15 1:07 PM, Cantor, Scott wrote:
>>
>> I assume that setting disregardTlsCertificate also works around this if the TLS part is a formality?
>
>Yes, it should work around, since that wires a null HostnameVerifier
>and the verification is therefore completely skipped.

I assumed, I just can't bring myself to trust(sic) myself on this stuff anymore.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
In reply to this post by Nate Klingenstein
On 7/23/15, 1:11 PM, "users on behalf of Nate Klingenstein" <[hidden email] on behalf of [hidden email]> wrote:

>This does blow up a lot of services I’m running which are not tethered to specific IP addresses at all, so it’s going to be really fun for me personally.

Well a) we have always said this was a bad idea and we weren't wrong, and b) at least this only affects people who have access to the latest Oracle code or are on Java 8, and that's a small minority.

Unless OpenJDK pushed this same bug and RH picked it up.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Nate Klingenstein
Well a) we have always said this was a bad idea and we weren't wrong

That’s fine, but every large deployment I know of has quite a bit of IP address “intelligence” built into DNS resolution to scale.  AWS ELB is my specific challenge here, but you could pick basically any “cloud” hosting service.

The LDAP issue Brent found is going to be even more fun if that proves to be an issue, and I don’t think ldaps has been against any recommendations.



--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Cantor, Scott E.
On 7/23/15, 1:27 PM, "users on behalf of Nate Klingenstein" <[hidden email] on behalf of [hidden email]> wrote:

>That’s fine, but every large deployment I know of has quite a bit of IP address “intelligence” built into DNS resolution to scale.  AWS ELB is my specific challenge here, but you could pick basically any “cloud” hosting service.

I don't understand what that has to do with this bug. Resolving the name to do a check is itself the bug.

>The LDAP issue Brent found is going to be even more fun if that proves to be an issue, and I don’t think ldaps has been against any recommendations.

Again, I don't see this affecting a majority of sites or even a significant minority unless RH pushes the fix. Java 8 adoption is quite low, and most people don't pay Oracle for updates. We should have a window to get a fix out.

Or of course there's the radical idea that maybe Oracle should fix their own bug and not rely on us to work around it?

And there's a trivial flag you can set on the JVM to work around it too. As problems go, I can think of many worse ones.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Shib IdP - Metadata Download and Java 1.7.0_85

Brent Putman
In reply to this post by Cantor, Scott E.


On 7/23/15 1:19 PM, Cantor, Scott wrote:
> Yes, it should work around, since that wires a null HostnameVerifier
> and the verification is therefore completely skipped.
> I assumed, I just can't bring myself to trust(sic) myself on this stuff anymore.

:-)  I actually just tested it, and it works with
disregardTlsCertificate=true.

--
To unsubscribe from this list send an email to [hidden email]
123