preloading high-value entity metadata

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

preloading high-value entity metadata

Tom Scavo
I added Example 4 to the ChainingMetadataProvider topic. [1] The
example shows how to preload "high-value SP metadata" using one or
more providers of type FileBackedHTTPMetadataProvider. Given the
robustness of a FileBackedHTTPMetadataProvider, this is a natural
thing to do. I know of at least one federation that explicitly
recommends such a strategy.

Taking this to its logical conclusion, we can predict: using
FileBackedHTTPMetadataProvider for "high-value SPs" will not scale in
the same way that FilesystemMetadataProvider does not scale for local
metadata. In going from Example #1 to Example #2 in the wiki, multiple
providers of type FilesystemMetadataProvider are replaced by a single
LocalDynamicMetadataProvider (which is a Good Thing). We need
something similar as we go from Example #3 to Example #4 (if we go
that way at all).

Consider the following hypothetical situation. Suppose there were a
provider called DirectoryBackedHTTPMetadataProvider:

- The provider has an attribute called backingDir, which is a
directory containing high-value entity metadata.

- As a reloading provider, the metadata in the backingDir is kept
fresh by periodically iterating over the backing files. In each case,
the metadata request URL is constructed by applying a transform to the
entityID (just like a dynamic provider).

- The reloading provider supports the same child elements as the
DynamicHTTPMetadataProvider.

So the provider is both a dynamic provider and a reloading provider.
Actually a new provider type is not strictly necessary, just add the
backingDir attribute to the DynamicHTTPMetadataProvider type. Whenever
a new entityID is encountered, spool a copy of entity metadata into
the backingDir.

I intentionally skipped all the details to keep this brief. Does it
make sense to extend DynamicHTTPMetadataProvider in this way?

Tom

[1] https://wiki.shibboleth.net/confluence/x/NgInAQ
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Cantor, Scott E.
> I intentionally skipped all the details to keep this brief. Does it make sense to
> extend DynamicHTTPMetadataProvider in this way?

I thought it already did all that, that was one of the features identified by the original InCommon WG, the ability to back up the query results and reload them up front to prime the cache. I admittedly don't know how to configure it. Maybe it got missed or just wasn't documented.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Rod Widdowson
> I thought it already did all that, that was one of the features identified by the original InCommon WG, the ability to back up the
query
> results and reload them up front to prime the cache. I admittedly don't know how to configure it. Maybe it got missed or just
wasn't
> documented.
>

If I follow this thread (I've been elsewhere) you need the
        persistentCacheManagerRef
        persistentCacheManagerDirectory
        persistentCacheKeyGeneratorRef
        initializeFromPersistentCacheInBackground
        initializationFromCachePredicateRef
        backgroundInitializationFromCacheDelay

attributes.  As per [1].  This gives you control of how, where, and what is cached and what and when is reloaded.

See [2]


[1]
https://wiki.shibboleth.net/confluence/display/IDP30/MetadataConfiguration#MetadataConfiguration-DynamicAttributesDynamicAttributes

[2]
http://git.shibboleth.net/view/?p=java-identity-provider.git;a=blob_plain;f=idp-profile-spring/src/test/resources/net/shibboleth/idp
/profile/spring/relyingparty/metadata/dynamicPersistentCacheDirectory.xml;h=44ef5a0c77ce28905f6ef7d74b8d23574f27a0d1;hb=HEAD

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Cantor, Scott E.
> If I follow this thread (I've been elsewhere) you need the
> persistentCacheManagerRef
> persistentCacheManagerDirectory
> persistentCacheKeyGeneratorRef
> initializeFromPersistentCacheInBackground
> initializationFromCachePredicateRef
> backgroundInitializationFromCacheDelay
>
> attributes.  As per [1].  This gives you control of how, where, and what is
> cached and what and when is reloaded.

Yeah, that's it.

I don't like having the settings split across multiple pages for all the reasons noted, but I think the more important issue is just for the main page to provide explicit pointers and examples on how to configure the basic features (this being one).

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: preloading high-value entity metadata

Tom Scavo
On Wed, May 16, 2018 at 12:46 PM, Cantor, Scott <[hidden email]> wrote:
>
> I don't like having the settings split across multiple pages for all the reasons noted, but I think the more important issue is just for the main page to provide explicit pointers and examples on how to configure the basic features (this being one).

Rod doesn't like it either so I will try to refactor the pages using
the include macro.

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: preloading high-value entity metadata

Tom Scavo
In reply to this post by Cantor, Scott E.
On Wed, May 16, 2018 at 12:46 PM, Cantor, Scott <[hidden email]> wrote:

>> If I follow this thread (I've been elsewhere) you need the
>>       persistentCacheManagerRef
>>       persistentCacheManagerDirectory
>>       persistentCacheKeyGeneratorRef
>>       initializeFromPersistentCacheInBackground
>>       initializationFromCachePredicateRef
>>       backgroundInitializationFromCacheDelay
>>
>> attributes.  As per [1].  This gives you control of how, where, and what is
>> cached and what and when is reloaded.
>
> Yeah, that's it.

Yes, I know about the persistent cache but that's not what I was
asking about. AFAIK, that cache only affects the startup process (like
the backing file on FileBackedHTTPMetadataProvider).

Please take a look at Example 4 on the ChainingMetadataProvider page.
[1] That example uses FileBackedHTTPMetadataProvider to short-circuit
a DynamicHTTPMetadataProvider. Is that something we want to document,
and if so, how can it be generalized? (I'm pretty sure the answer to
the latter question is that it can't be generalized, at least not with
what we have available to us now.)

Thanks,

Tom

[1] https://wiki.shibboleth.net/confluence/x/NgInAQ
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Cantor, Scott E.
> AFAIK, that cache only affects the startup process (like the backing file on
> FileBackedHTTPMetadataProvider).

I don't know what else you're trying to do.

> Please take a look at Example 4 on the ChainingMetadataProvider page.
> [1] That example uses FileBackedHTTPMetadataProvider to short-circuit a
> DynamicHTTPMetadataProvider.

That doesn't really make any sense. If you can use an aggregate, you're done. There's no point in doing both. A File-based metadata source locally might be an alternative to the cache approach, that's always been the usual way of hijacking a remote source because you need to correct for an error or something. That can be a batch file or LocalDynamic, whichever. It's always a good idea to have that hook, but most people already have local metadata installed, and yes, it's usually best if it's first.

> Is that something we want to document, and
> if so, how can it be generalized? (I'm pretty sure the answer to the latter
> question is that it can't be generalized, at least not with what we have available
> to us now.)

It makes no sense to me at all. If you're beta testing a dynamic source or something like that, you'd do the opposite, put the aggregate source after the dynamic lookup as a fallback.

The problem right now is that Dynamic can't run anywhere but the end of a chain because it doesn't cache failure. Until that's cleaned up it has to be used very carefully or it will take down your system performance-wise.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Cantor, Scott E.
> The problem right now is that Dynamic can't run anywhere but the end of a
> chain because it doesn't cache failure. Until that's cleaned up it has to be used
> very carefully or it will take down your system performance-wise.

By which I just mean that's the one bit of subtlety to be careful of, it doesn't really have anything to do with the original question per se.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: preloading high-value entity metadata

Tom Scavo
In reply to this post by Cantor, Scott E.
On Wed, May 16, 2018 at 1:37 PM, Cantor, Scott <[hidden email]> wrote:
>
>> Please take a look at Example 4 on the ChainingMetadataProvider page.
>> [1] That example uses FileBackedHTTPMetadataProvider to short-circuit a
>> DynamicHTTPMetadataProvider.
>
> That doesn't really make any sense. If you can use an aggregate, you're done.

There is no aggregate being consumed in Example 4 so that tells me the
example is unclear. I'll fix that but in the meantime can you skim
this federation doc page?

https://www.ukfederation.org.uk/content/Documents/MDQ

I think you'll get the idea fairly quickly without having to wade
through all the details.

Thanks,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Cantor, Scott E.
> I think you'll get the idea fairly quickly without having to wade through all the
> details.

Our decision was that if you want to pre-fetch, you can do that yourself (just tell the IdP to do it as part of startup scripts and warn you). I added a metadata query flow to 3.4 so it doesn't take much scripting to hit it with a few entityIDs to prime and send alerts if it doesn't have metadata on hand for that system. Prior, you'd just mock Unsolicited requests I imagine, but that's why I added the feature.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: preloading high-value entity metadata

Tom Scavo
On Wed, May 16, 2018 at 2:17 PM, Cantor, Scott <[hidden email]> wrote:
>> I think you'll get the idea fairly quickly without having to wade through all the
>> details.
>
> Our decision was that if you want to pre-fetch, you can do that yourself (just tell the IdP to do it as part of startup scripts and warn you). I added a metadata query flow to 3.4 so it doesn't take much scripting to hit it with a few entityIDs to prime and send alerts if it doesn't have metadata on hand for that system.

Is this new flow documented?

In any case, I've removed Example 4 from the ChainingMetadataProvider
page until all of this shakes out a bit more.

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: preloading high-value entity metadata

Cantor, Scott E.
> Is this new flow documented?

Doubt it, I was just taking small breaks from the SP and didn't spend much time on it.

It's just /idp/profile/admin/mdquery?saml2&entityID=

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: preloading high-value entity metadata

Tom Scavo
On Wed, May 16, 2018 at 2:56 PM, Cantor, Scott <[hidden email]> wrote:
>> Is this new flow documented?
>
> Doubt it, I was just taking small breaks from the SP and didn't spend much time on it.
>
> It's just /idp/profile/admin/mdquery?saml2&entityID=

Okay, that's cool. I won't spend any more time on this. Let's see how
it shakes out.

Tom
--
To unsubscribe from this list send an email to [hidden email]