SP V3 metadata types

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

SP V3 metadata types

Tom Scavo
I was looking over the SP V3 metadata docs and noticed two features
that I wish the IdP had: the Folder type and the MDQ type. (I don't
like the names but that's beside the point.) The latter will become a
useful convenience for deployers. Despite the appearance of
LocalDynamicMetadataProvider, the former will become useful if the
notion of "high-value SP" catches on. I'm betting that it will.

Just my two cents,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP V3 metadata types

Cantor, Scott E.
> I was looking over the SP V3 metadata docs and noticed two features that I
> wish the IdP had: the Folder type and the MDQ type. (I don't like the names but
> that's beside the point.) The latter will become a useful convenience for
> deployers. Despite the appearance of LocalDynamicMetadataProvider, the
> former will become useful if the notion of "high-value SP" catches on. I'm
> betting that it will.

Folder is all but deprecated (probably will be, I haven't decided yet), given how it's implemented it's useless now. It's been replaced by LocalDynamic, which is superior. The file name issue is annoying but it's just necessary to make it work well.

AFAIK, the syntax for the IdP is equivalent to the MDQ shorthand we added, so I don't think that's really much different.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
On Mon, Jun 11, 2018 at 12:54 PM, Cantor, Scott <[hidden email]> wrote:
>> I was looking over the SP V3 metadata docs and noticed two features that I
>> wish the IdP had: the Folder type and the MDQ type. (I don't like the names but
>> that's beside the point.) The latter will become a useful convenience for
>> deployers. Despite the appearance of LocalDynamicMetadataProvider, the
>> former will become useful if the notion of "high-value SP" catches on. I'm
>> betting that it will.
>
> Folder is all but deprecated (probably will be, I haven't decided yet), given how it's implemented it's useless now.

I don't know anything about the implementation.

> It's been replaced by LocalDynamic, which is superior.

For the use cases you have in mind, yes, that's probably true. There
is another use case emerging, called the "high-value SP," and AFAICT
the LocalDynamicMetadataProvider type is of no use in that case.

> AFAIK, the syntax for the IdP is equivalent to the MDQ shorthand we added

Hmm, if that's true, I don't know about it. It is certainly not documented.

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Peter Schober
* Tom Scavo <[hidden email]> [2018-06-11 19:03]:
> > Folder is all but deprecated (probably will be, I haven't decided yet), given how it's implemented it's useless now.
>
> I don't know anything about the implementation.

IIRC it only loads the files present at startup and doesn't "learn"
about newly added ones once the process runs. I.e., useless.
-peter
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP V3 metadata types

Cantor, Scott E.
> IIRC it only loads the files present at startup and doesn't "learn"
> about newly added ones once the process runs. I.e., useless.

And incredibly inefficient on large directories.

I have no idea what use case could possibly be met by this that LocalDynamic doesn't handle, with the downside of the filename thing, but that's not that big a deal.

"Importance" is not a local/remote consideration, we handled "important" by dealing with reliability, making sure we have proper caching, and the ability to cache them across restarts and such. I have lots of mission-critical SPs that we rely on InCommon for. If that didn't work, obviously InCommon would be in trouble.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
On Mon, Jun 11, 2018 at 1:28 PM, Cantor, Scott <[hidden email]> wrote:
>
> "Importance" is not a local/remote consideration, we handled "important" by dealing with reliability, making sure we have proper caching, and the ability to cache them across restarts and such. I have lots of mission-critical SPs that we rely on InCommon for. If that didn't work, obviously InCommon would be in trouble.

AFAIK, InCommon has not yet exposed a production metadata query
server. The use case I have in mind works in conjunction with MDQ.

I previously posted the following pointer:
https://www.ukfederation.org.uk/content/Documents/MDQ

The above documentation page defines (by example) the notion of
"high-value entity." Please review this external doc so we can be on
the same page.

I hesitated posting this again since it may be misinterpreted. I am
not making a value judgment. I am simply pointing out that the largest
federation in the world is making this recommendation. If you step
back and think about it, it's actually quite reasonable to do. Whether
it catches on worldwide is another question.

Btw, I've added a similar example to the wiki:
https://wiki.shibboleth.net/confluence/x/YwbKAg

If you (the Shib dev team) don't think this is a Good Thing To Do,
please let me know so I can remove the example from the wiki.

Thanks,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP V3 metadata types

Cantor, Scott E.
> I hesitated posting this again since it may be misinterpreted. I am not making a
> value judgment. I am simply pointing out that the largest federation in the
> world is making this recommendation. If you step back and think about it, it's
> actually quite reasonable to do. Whether it catches on worldwide is another
> question.

We don't think anything like that is really needed, but to the extent that you want to do it we simply saw it as something you can do by priming the software at startup with the necessary queries, it doesn't need to be explicitly configured. It also presumes you actually restart your IdP frequently, and no mature IdP does if it's operated the way we intend.
 
Either way, the Folder hack doesn't really work for that, so I don't know what you had in mind there. It doesn't load fragmentary metadata configs, but actual metadata outright, so it would subsume the remote copy.
 
> If you (the Shib dev team) don't think this is a Good Thing To Do, please let me
> know so I can remove the example from the wiki.

I believe we felt it unnecessary in light of existing features and how it's all meant to be run. But to the extent you want to do it, the SP already can (as shown there) and Folder certainly doesn't help.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
On Mon, Jun 11, 2018 at 2:13 PM, Cantor, Scott <[hidden email]> wrote:
>
> ... the Folder hack doesn't really work for that, so I don't know what you had in mind there.

What happens if you take the notion of "high-value entity" to its
logical conclusion? The concept is really quite simple. Suppose an IdP
has access to a trusted, remote metadata aggregate containing most of
its SP partners. (That's a big "IF" but please bear with me as I make
my point.) In that case, the tail of the provider chain would consist
of a FileBackedHTTPMetadataProvider (to load the aggregate) followed
by a DynamicHTTPMetadataProvider (to pick up any stray SPs not in the
aggregate). QED

Such a configuration minimizes the number of times the
DynamicHTTPMetadataProvider makes a request. If it does fail, it fails
on an SP that is not a recognized SP partner (since its not in the
aggregate).

As a hypothetical concept, does that make sense?

>> If you (the Shib dev team) don't think this is a Good Thing To Do, please let me
>> know so I can remove the example from the wiki.
>
> I believe we felt it unnecessary in light of existing features and how it's all meant to be run.

I was asking about the IdP documentation...if anyone on the dev team
thinks the wiki example [1] is counterproductive, I will remove it.

Tom

[1] RemoteEntityMetadataExample https://wiki.shibboleth.net/confluence/x/YwbKAg
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP V3 metadata types

Cantor, Scott E.
> What happens if you take the notion of "high-value entity" to its logical
> conclusion? The concept is really quite simple. Suppose an IdP has access to a
> trusted, remote metadata aggregate containing most of its SP partners. (That's
> a big "IF" but please bear with me as I make my point.) In that case, the tail of
> the provider chain would consist of a FileBackedHTTPMetadataProvider (to
> load the aggregate) followed by a DynamicHTTPMetadataProvider (to pick up
> any stray SPs not in the aggregate). QED

Sure, though obviously that presupposes that file is something you can come up with without just pointing at the aggregate and ending up with a 5 minute startup and 1G of RAM. It would make sense if federations provided a custom aggregate capability, but otherwise I don't see it really happening.

> Such a configuration minimizes the number of times the
> DynamicHTTPMetadataProvider makes a request. If it does fail, it fails on an SP
> that is not a recognized SP partner (since its not in the aggregate).
>
> As a hypothetical concept, does that make sense?

Sure, in theory, just not really in practice today.

> I was asking about the IdP documentation...if anyone on the dev team thinks
> the wiki example [1] is counterproductive, I will remove it.

I think it's probably not crazy to show in the IdP today, but I've already asked Brent to improve some of the behavior of the dynamic plugin and I think that will render it unnecessary so I would prefer we not complicate examples with it now.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
On Mon, Jun 11, 2018 at 3:01 PM, Cantor, Scott <[hidden email]> wrote:
>>
>> As a hypothetical concept, does that make sense?
>
> Sure, in theory, just not really in practice today.

I agree, custom remote metadata aggregates are not very practical
today. OTOH, a DynamicHTTPMetadataProvider aggregates the metadata of
your SP partners as a side effect. The current implementation doesn't
fully leverage that metadata, however.

The current implementation of DynamicHTTPMetadataProvider caches
entity metadata in a local directory (if
persistentCacheManagerDirectory is specified) but then the cache is
treated as nothing more than a glorified backing file (at least
AFAICT). The provider needs to be smarter about how it leverages
cached metadata.

At this point, I'm tempted to dive into a number of "what ifs" but
that would probably be premature since (as you say) some modifications
are coming. I hope I'm watching the relevant jira issues. Can/should
we discuss it here instead?

>> I was asking about the IdP documentation...if anyone on the dev team thinks
>> the wiki example [1] is counterproductive, I will remove it.
>
> I think it's probably not crazy to show in the IdP today, but I've already asked Brent to improve some of the behavior of the dynamic plugin and I think that will render it unnecessary so I would prefer we not complicate examples with it now.

Okay, the example is gone. I still think our story surrounding remote
metadata is incomplete.

Thanks,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: SP V3 metadata types

Cantor, Scott E.
> The current implementation of DynamicHTTPMetadataProvider caches entity
> metadata in a local directory (if persistentCacheManagerDirectory is specified)
> but then the cache is treated as nothing more than a glorified backing file (at
> least AFAICT). The provider needs to be smarter about how it leverages cached
> metadata.

I agree, but I'm not sure your idea of "smarter" matches mine, I just asked for some particular caching and change detection improvements and I think they're underway. They aren't attempts to turn "remote" into "local". People don't cache DNS that way, and this is the same, you either trust a remote source or you're paranoid (possibly for good reason) and want to do it all yourself, in which case you can just script up your own folder to taste and use LocalDynamic. I don't think there's sufficient demand for us to bother with that. I think we implemented (or have in progress) all the improvements that were suggested by the InCommon WG).

> At this point, I'm tempted to dive into a number of "what ifs" but that would
> probably be premature since (as you say) some modifications are coming. I
> hope I'm watching the relevant jira issues. Can/should we discuss it here
> instead?

JIRA isn't good for long side discussions, so if there's something not in JIRA that you think you want done, I would take it up here with Brent, because I'm not going to be the one implementing it anyway. The issues I had I filed and they're fairly limited in scope.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Brent Putman
In reply to this post by Tom Scavo



On 6/11/18 3:46 PM, Tom Scavo wrote:

The current implementation of DynamicHTTPMetadataProvider caches
entity metadata in a local directory (if
persistentCacheManagerDirectory is specified) but then the cache is
treated as nothing more than a glorified backing file (at least
AFAICT). The provider needs to be smarter about how it leverages
cached metadata.

Like Scott, I don't think we agree on what "smarter" could mean.  Are you perhaps missing that the metadata in the persistent cache is loaded into memory on every refresh? (as long as still valid, not expired, etc)  What that means is that, once it's primed into the provider by the first request for it [1], it's going to effectively be available for as long as it is still valid - the same as an aggregate would be.

At that point it effectively is "local", in that no query is attempted for those entities, and it doesn't matter whether the source server is available etc.  Isn't that what you're essentially trying to do with the notion of the separate batch provider accessing an aggregate of "high value" entities?

[1] As far as priming the dynamic providers, Scott mentioned the use of a script to issue requests.  I can't remember whether we built an admin endpoint into the metadata service to do that (as opposed to just issuing IdP-initiated requests whose responses you just disregard).  We could.  But the other approach we talked about was a simple Spring bean that would take a metadata resolver ID and a list of entityIDs and just do the resolution/priming internally on IdP (re)start.


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Cantor, Scott E.
On 6/11/18, 4:57 PM, "dev on behalf of Brent Putman" <[hidden email] on behalf of [hidden email]> wrote:

> [1] As far as priming the dynamic providers, Scott mentioned the use of a script to issue requests.  I can't remember
> whether we built an admin endpoint into the metadata service to do that

I did by inference, there's an admin endpoint and a command line script to query metadata, and that would prime the system (and it returns status codes so it's possible to detect failure and do "something", whatever that would be).

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
In reply to this post by Cantor, Scott E.
On Mon, Jun 11, 2018 at 3:57 PM, Cantor, Scott <[hidden email]> wrote:
>
> I'm not sure your idea of "smarter" matches mine, I just asked for some particular caching and change detection improvements and I think they're underway. They aren't attempts to turn "remote" into "local". People don't cache DNS that way, and this is the same, you either trust a remote source or you're paranoid (possibly for good reason) and want to do it all yourself, in which case you can just script up your own folder to taste and use LocalDynamic. I don't think there's sufficient demand for us to bother with that. I think we implemented (or have in progress) all the improvements that were suggested by the InCommon WG).

I don't agree with all of that, but in any case, I'm beginning to get
your drift. As a compromise, I added a couple of "TIPS" to the section
on Remote Metadata on the MetadataManagementBestPractices page. [1]
The tip in the DynamicHTTPMetadataProvider section reads as follows:

<tip>
Ask your federation operator how best to configure a metadata provider
of type DynamicHTTPMetadataProvider. In particular, determine the base
URL of the MDQ server, Also ask about the recommended values of the
minCacheDuration attribute (default: PT10M) and the maxCacheDuration
attribute (default: PT8H). Finally, ask how best to configure the
provider to mitigate the risk of an MDQ server that is unreachable or
nonresponsive.
</tip>

I'm guessing that's closer to what you had in mind, that is, the
deployer should be advised to consult the federation operator.

Comments welcome.

Tom

[1] MetadataManagementBestPractices
https://wiki.shibboleth.net/confluence/x/JQXKAg
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
In reply to this post by Brent Putman
Brent, thanks for weighing in.

On Mon, Jun 11, 2018 at 4:57 PM, Brent Putman <[hidden email]> wrote:
>
> Like Scott, I don't think we agree on what "smarter" could mean.

No doubt about that :-)

> Are you
> perhaps missing that the metadata in the persistent cache is loaded into
> memory on every refresh?

Can you define what you mean by "refresh" here? Do you mean query?
(Sorry, I don't mean to be overly pedantic.)

> (as long as still valid, not expired, etc)  What
> that means is that, once it's primed into the provider by the first request
> for it [1], it's going to effectively be available for as long as it is
> still valid - the same as an aggregate would be.

Okay, but under what conditions will the persistent cache be accessed
by the IdP?

> At that point it effectively is "local", in that no query is attempted for
> those entities, and it doesn't matter whether the source server is available
> etc.  Isn't that what you're essentially trying to do with the notion of the
> separate batch provider accessing an aggregate of "high value" entities?

Yes, I think so, but I'm not sure since this is new to me. I've read
all the documentation and AFAIK there is nothing in the docs along
these lines. What did I miss?

In any case, I'm not sure I understand this well enough to document
it. Can you describe what happens if the server goes down for, say, 20
mins?

Also, what happens if the server returns 304 Not Modified?

> ...
> the other approach we talked about was a simple Spring bean that would take
> a metadata resolver ID and a list of entityIDs and just do the
> resolution/priming internally on IdP (re)start.

Is this another application of <md:AffiliationDescriptor>?

Thanks Brent.

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Brent Putman



On 6/12/18 10:01 AM, Tom Scavo wrote:


Can you define what you mean by "refresh" here? Do you mean query?
(Sorry, I don't mean to be overly pedantic.)

No, not query.  Actually I think I meant to say "reload" not refresh.  I meant when the metadata resolver service or a specific resolver is (re)loaded, either on IdP (re)start, or explicitly through use of the the admin endpoints.


Okay, but under what conditions will the persistent cache be accessed
by the IdP?

Well, it's only *read* when the resolver is initialized/restarted. (A *write* happens whenever data is resolved from the source).  But whatever is in the cache is by definition also "live" and in-memory, since it only got into the cache by coming in through the resolver.  So the persistent cache is just there to re-init the state of the resolver on a restart.  If you have 150 live valid entries before restart, you'll have 150 live valid entries after restart (generally, that's not *exactly* correct always, but I won't go into the edge cases).



At that point it effectively is "local", in that no query is attempted for
those entities, and it doesn't matter whether the source server is available
etc.  Isn't that what you're essentially trying to do with the notion of the
separate batch provider accessing an aggregate of "high value" entities?
Yes, I think so, but I'm not sure since this is new to me. I've read
all the documentation and AFAIK there is nothing in the docs along
these lines. What did I miss?

If an entry is live/in-memory and valid, a source query (e.g. HTTP request) will not be issued until you start hitting the effective "next refresh" time for that entity (which as in the batch resolvers is based on a combination of entity @validUntil and @cacheDuration plus resolver config).   So once the resolver is "primed" with an entity, requests for it get serviced from the in-memory cache.  No request to the source will be made (until it's time to refresh based on expiration and caching info).


In any case, I'm not sure I understand this well enough to document
it. Can you describe what happens if the server goes down for, say, 20
mins?

I think there's 3 cases:

1) If the metadata for the entity in question is still live and valid and isn't past the point where the "next refresh" kicks in, then no HTTP request is issued, so the IdP won't even know the metadata server is down.

2) If the metadata *is* past the next refresh boundary, then an attempt is made to re-fetch from the source. If it works, great, the live metadata is updated accordingly. If it doesn't work as in your question (e.g. HTTP client times out, etc), then nothing gets updated in the resolver.  However, if the live metadata is still valid, then it will continue to be used.  That's why we start trying to refresh the metadata *before* its validity/cache expiration, via applying the refreshDelayFactor.

3) If the server is down and the refresh attempt fails and the metadata in-memory is expired/invalid, then no entity is returned from that resolver.


Also, what happens if the server returns 304 Not Modified?

Well, first off, you'd only see that if the HttpClient instance in use was of the caching variety.  It's the HttpClient that would decide to issue a conditional GET, not the resolver.  As discussed extensively in another thread, the dynamic HTTP resolver code itself does not cache/utilize Last-Modified and ETag nor does it do conditional GET internally; it delegates all that to the HttpClient instance.

So, the dynamic HTTP resolver itself will not really see or handle any 304, it's a matter for the HttpClient instance.  If the latter is of the caching variety and it issues a conditional GET and gets a 304, then it's going to return the previously cached HTTP response to the resolver.  The metadata resolver itself won't know or care whether the HTTP response came from a "real" fetch that returned a 200 + data, vs a cached response from a conditional GET that resulted in a 304 (vs for the record a request that was serviced entirely from HTTP cache (no HTTP requests at all) based on the presence of an appropriate Cache-Control response header in a previous response).


...
the other approach we talked about was a simple Spring bean that would take
a metadata resolver ID and a list of entityIDs and just do the
resolution/priming internally on IdP (re)start.
Is this another application of <md:AffiliationDescriptor>?

Well, I've never really thought of it that way personally.  I see the priming as more of a local config thing, and AffiliationDescriptor as describing functional collections of entities for SAML purposes (VOs, etc).  I guess it theoretically could be used to drive the priming.  I think there's an implicit assumption in there though that the AffiliationDescriptor's members are all in that particular batch of metadata.  And I don't think MDQ is even currently defined to return AffiliationDescriptor is it?  So then it wouldn't even be relevant to the dynamic case, only batch.

However, since Scott built in the admin endpoint to do the priming via simple curl script, then I doubt we'd ever actually add the bean I described.


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
On Tue, Jun 12, 2018 at 2:16 PM, Brent Putman <[hidden email]> wrote:

>>
>> On 6/12/18 10:01 AM, Tom Scavo wrote:
>>
>> In any case, I'm not sure I understand this well enough to document
>> it. Can you describe what happens if the server goes down for, say, 20
>> mins?
>
> I think there's 3 cases:
>
> 1) If the metadata for the entity in question is still live and valid and
> isn't past the point where the "next refresh" kicks in, then no HTTP request
> is issued, so the IdP won't even know the metadata server is down.

Do minCacheDuration and maxCacheDuration have an effect at this level
or the HttpCache level?

> 2) If the metadata *is* past the next refresh boundary, then an attempt is
> made to re-fetch from the source. If it works, great, the live metadata is
> updated accordingly. If it doesn't work as in your question (e.g. HTTP
> client times out, etc), then nothing gets updated in the resolver.  However,
> if the live metadata is still valid, then it will continue to be used.
> That's why we start trying to refresh the metadata *before* its
> validity/cache expiration, via applying the refreshDelayFactor.

Is the above behavior independent of the persistent cache? (I'm
guessing the answer is yes.)

> 3) If the server is down and the refresh attempt fails and the metadata
> in-memory is expired/invalid, then no entity is returned from that resolver.

Yes, that makes sense.

>> Also, what happens if the server returns 304 Not Modified?
>
> Well, first off, you'd only see that if the HttpClient instance in use was
> of the caching variety.

Which it is by default, right?

> It's the HttpClient that would decide to issue a
> conditional GET, not the resolver.  As discussed extensively in another
> thread, the dynamic HTTP resolver code itself does not cache/utilize
> Last-Modified and ETag nor does it do conditional GET internally; it
> delegates all that to the HttpClient instance.

Yes, I understand that now.

> So, the dynamic HTTP resolver itself will not really see or handle any 304,
> it's a matter for the HttpClient instance.  If the latter is of the caching
> variety and it issues a conditional GET and gets a 304, then it's going to
> return the previously cached HTTP response to the resolver.  The metadata
> resolver itself won't know or care whether the HTTP response came from a
> "real" fetch that returned a 200 + data, vs a cached response from a
> conditional GET that resulted in a 304 (vs for the record a request that was
> serviced entirely from HTTP cache (no HTTP requests at all) based on the
> presence of an appropriate Cache-Control response header in a previous
> response).

Right, that agrees with what Scott and Rod told me earlier, but that
is suboptimal since the metadata resolver redundantly processes the
metadata in the case of 304. That processing includes signature
verification and who knows whet other metadata filters have been
configured on the DynamicHTTPMetadataProvider. I'm not intimately
familiar with the implementation but there seems to be a significant
optimization lurking here, essentially the same optimization
implemented in FileBackedHTTPMetadataProvider.

Thanks for taking the time to answer my naive questions.

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Cantor, Scott E.
On 6/13/18, 9:33 AM, "dev on behalf of Tom Scavo" <[hidden email] on behalf of [hidden email]> wrote:

> Right, that agrees with what Scott and Rod told me earlier, but that
> is suboptimal since the metadata resolver redundantly processes the
> metadata in the case of 304. That processing includes signature
> verification and who knows whet other metadata filters have been
> configured on the DynamicHTTPMetadataProvider. I'm not intimately
> familiar with the implementation but there seems to be a significant
> optimization lurking here, essentially the same optimization
> implemented in FileBackedHTTPMetadataProvider.

Yes, exactly, that's one of the enhancements I asked for, the other being caching failure to avoid repeated failed lookups.

The same thing happens with LocalDynamic. Even if the local file is unchanged, it will reload it every time it tries, and go through the full processing step when it does.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Brent Putman
In reply to this post by Tom Scavo



On 6/13/18 9:33 AM, Tom Scavo wrote:

1) If the metadata for the entity in question is still live and valid and
isn't past the point where the "next refresh" kicks in, then no HTTP request
is issued, so the IdP won't even know the metadata server is down.
Do minCacheDuration and maxCacheDuration have an effect at this level
or the HttpCache level?

Yes, those 2 attributes control the behavior around when the resolver attempts the "next refresh" (in addition to @validUntil and @cacheDuration in the metadata). 

They do not have anything to do with the caching layer of HttpClient.


      
2) If the metadata *is* past the next refresh boundary, then an attempt is
made to re-fetch from the source. If it works, great, the live metadata is
updated accordingly. If it doesn't work as in your question (e.g. HTTP
client times out, etc), then nothing gets updated in the resolver.  However,
if the live metadata is still valid, then it will continue to be used.
That's why we start trying to refresh the metadata *before* its
validity/cache expiration, via applying the refreshDelayFactor.
Is the above behavior independent of the persistent cache? (I'm
guessing the answer is yes.)

Yes, persistent cache has nothing to do with this.


      
Also, what happens if the server returns 304 Not Modified?
Well, first off, you'd only see that if the HttpClient instance in use was
of the caching variety.
Which it is by default, right?

Yes, the IdP dynamic HTTP resolver defaults to the 'memory' caching HttpClient instance.


      
Right, that agrees with what Scott and Rod told me earlier, but that
is suboptimal since the metadata resolver redundantly processes the
metadata in the case of 304. That processing includes signature
verification and who knows whet other metadata filters have been
configured on the DynamicHTTPMetadataProvider. I'm not intimately
familiar with the implementation but there seems to be a significant
optimization lurking here, essentially the same optimization
implemented in FileBackedHTTPMetadataProvider.

Will respond to this in the next message.

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: SP V3 metadata types

Tom Scavo
On Thu, Jun 14, 2018 at 3:32 PM, Brent Putman <[hidden email]> wrote:

>>
>> On 6/13/18 9:33 AM, Tom Scavo wrote:
>>
>> Also, what happens if the server returns 304 Not Modified?
>
> Well, first off, you'd only see that if the HttpClient instance in use was
> of the caching variety.
>
>> Which it is by default, right?
>
> Yes, the IdP dynamic HTTP resolver defaults to the 'memory' caching
> HttpClient instance.

Yes, that's what I thought. Thanks for confirming.

Related to this, will you consider deprecating the so-called HTTP
Caching Attributes on the HTTP metadata providers?

https://issues.shibboleth.net/jira/browse/IDP-1287

Thanks,

Tom
--
To unsubscribe from this list send an email to [hidden email]
12