Best Practices for Metadata Management

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Best Practices for Metadata Management

Tom Scavo
At one point Scott suggested we might need a guide to best practices
for metadata management, something that summarized the uses and
advantages of various metadata providers, at least for IdP deployers.
(That's my recollection anyway, I can't find the original comment.)

I created a new document in the wiki that outlines a set of best
practices for IdP deployers. [1] Here's a brief critique of my own
work:

- Thanks to LocalDynamicMetadataProvider, I think we have a pretty
good story for local metadata. (Whether or not I hit the mark is an
open question.) All we need now are tools for deployers.

- The story surrounding remote metadata is incomplete. I don't know
what else to say. How do we mitigate the risk associated with
DynamicHTTPMetadataProvider?

- The use of entity attributes to drive automated relying party
configuration (a so-called metadata-driven configuration) is critical.
Scott has already written the definitive document on the subject. [2]

All comments are welcome of course.

Tom

[1] https://wiki.shibboleth.net/confluence/x/JQXKAg
[2] https://wiki.shibboleth.net/confluence/x/VQC_AQ
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Zeller-3
> All comments are welcome of course.

Maybe it would be useful in the "best practices" document to provide
(or link to) examples of how to add/remove/manage metadata for the
IdP, e.g. configuration snippets and commands/URLs to reload the
configuration. In other words, doc the best-practice plus how-to in
one place.

TomZ
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Scavo
On Tue, Jun 5, 2018 at 11:03 AM, Tom Zeller <[hidden email]> wrote:
>> All comments are welcome of course.
>
> Maybe it would be useful in the "best practices" document to provide
> (or link to) examples of how to add/remove/manage metadata for the
> IdP, e.g. configuration snippets and commands/URLs to reload the
> configuration. In other words, doc the best-practice plus how-to in
> one place.

Since the "best practices" page already contains links to specific
metadata provider pages, I assume you'd like to see examples inline.
That's a good idea, and now that I know how to do include pages, I
think that's doable for the most part. I'll try that for
FilesystemMetadataProvider and FileBackedHTTPMetadataProvider first
since the use of those providers is well understood. We'll go from
there.

Thanks,

TomS
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Scavo
On Tue, Jun 5, 2018 at 4:24 PM, Tom Scavo <[hidden email]> wrote:
>
> Since the "best practices" page already contains links to specific
> metadata provider pages, I assume you'd like to see examples inline.

I created a set of canonical examples [1] and included them inline in
the "best practices" page. [2]

TomZ, does this help? If you have other suggestions, let me know.

Note: There is no example of LocalDynamicMetadataProvider anywhere in
the wiki. Without tools, that would seem to be premature.

Tom

[1] MetadataProviderExamples https://wiki.shibboleth.net/confluence/x/8QXKAg
[2] MetadataManagementBestPractices
https://wiki.shibboleth.net/confluence/x/JQXKAg
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Best Practices for Metadata Management

Cantor, Scott E.
> Note: There is no example of LocalDynamicMetadataProvider anywhere in the
> wiki. Without tools, that would seem to be premature.

The metagen script I did for the SP that really needs to be moved into the IdP distro is pretty much the Swiss army knife of this sort of thing, it took me a few hours to whip up a "flat file of hosts -> local dynamic files" workflow based on a simple heuristic that fit the particular case I was dealing with (100 Drupal sites using SimpleSAML).

It's all still in flux, but it didn't take any new tools really, just tweaks and additions based on that one core script.

By the time 3.4 ships I'll make sure I get a reasonable updated version of it pushed into the IdP.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Scavo
On Thu, Jun 7, 2018 at 11:09 AM, Cantor, Scott <[hidden email]> wrote:
>
> The metagen script I did for the SP that really needs to be moved into the IdP distro is pretty much the Swiss army knife of this sort of thing,

Can you provide a pointer to this script and/or its documentation?

Thanks,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Best Practices for Metadata Management

Cantor, Scott E.
> Can you provide a pointer to this script and/or its documentation?

It's in the SP distribution, it's in etc/ like the keygen script is. Never been documented but because of it's nature it's a script you copy and modify for local use as much as anything else.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Zeller-3
In reply to this post by Tom Scavo
> TomZ, does this help? If you have other suggestions, let me know.

I'll take a look.

Another comment : do we use the word "register" to describe adding
metadata / trusting a relying-party ? If not, I suggest maybe we
should.

It seems like it's one thing to "join" a federation versus
"registering metadata with a federation" via "registering a
federation's metadata", and it would be nice to refer folks to a page
that describes what that means. Or maybe OIDC defines "register" for
us already.

Just some thoughts,
TomZ
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Scavo
On Thu, Jun 7, 2018 at 11:56 AM, Tom Zeller <[hidden email]> wrote:
>> TomZ, does this help? If you have other suggestions, let me know.
>
> I'll take a look.

Thanks.

> Another comment : do we use the word "register" to describe adding
> metadata / trusting a relying-party ?

No, that word is not used AFAIK.

> If not, I suggest maybe we should.
>
> It seems like it's one thing to "join" a federation versus
> "registering metadata with a federation" via "registering a
> federation's metadata"

I think the first two are out of scope. In the latter case, we show
the deployer how to "load federation metadata." See the section
"FileBackedHTTPMetadataProvider" in the "best practices" document.

> and it would be nice to refer folks to a page
> that describes what that means.

I'm not following you...can you be more specific?

> Or maybe OIDC defines "register" for
> us already.

This too is out of scope (at least for now).

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Zeller-3
>> and it would be nice to refer folks to a page
>> that describes what that means.
>
> I'm not following you...can you be more specific?

Sorry, sure.

For example, when talking to someone who has just stood up an IdP,
three common tasks are : join a federation, add/register the newly
stood up IdP's metadata to the federation metadata aggregate, as well
as add/register that aggregate to the newly stood up IdP as trusted
metadata.

Hope that makes sense,
Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Scavo
On Thu, Jun 7, 2018 at 12:23 PM, Tom Zeller <[hidden email]> wrote:
>
> ... when talking to someone who has just stood up an IdP,
> three common tasks are : join a federation, add/register the newly
> stood up IdP's metadata to the federation metadata aggregate, as well
> as add/register that aggregate to the newly stood up IdP as trusted
> metadata.
>
> Hope that makes sense

Yes it does, thanks. I added a TIP to the "Remote Metadata" section of
the "best practices" page. I took the opportunity to promote the
advantages of eduGAIN to Shibboleth IdP deployers.

Hope this helps,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for Metadata Management

Tom Scavo
In reply to this post by Cantor, Scott E.
On Thu, Jun 7, 2018 at 11:27 AM, Cantor, Scott <[hidden email]> wrote:
>> Can you provide a pointer to this script and/or its documentation?
>
> It's in the SP distribution, it's in etc/ like the keygen script is. Never been documented but because of it's nature it's a script you copy and modify for local use as much as anything else.

I found it:
http://git.shibboleth.net/view/?p=cpp-sp.git;a=blob;f=configs/metagen.sh;h=94e646ab0aad13e661f45861583eff866e0485b4;hb=HEAD

AFAICT, the primary purpose of this script is to create metadata
(hence, its name). In any case, I've been working on a complementary
script. It uses rsync to push files from one directory to another,
possibly on different hosts.

I'm trying to get my arms around the rsync command (which I'm not
familiar with). Can you post the command line you used to sync one
sourceDirectory with another while you were testing
LocalDynamicMetadataProvider on the IdP? (Yes, I know you're busy with
the SP right now. If this request needs to wait, that's fine.)

Thanks in advance,

Tom
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Best Practices for Metadata Management

Cantor, Scott E.
> AFAICT, the primary purpose of this script is to create metadata (hence, its
> name). In any case, I've been working on a complementary script. It uses rsync
> to push files from one directory to another, possibly on different hosts.

I have about three layers of scripts, and metagen is the one at the bottom. I have scripts that batch up calls to it, and then scripts that wrap all those calls in EntitiesDescriptor, and so forth. The dynamic stuff is new so I set that up for the first time when a customer handed me a flat file of 160 hosts.

> I'm trying to get my arms around the rsync command (which I'm not familiar
> with). Can you post the command line you used to sync one sourceDirectory
> with another while you were testing LocalDynamicMetadataProvider on the
> IdP? (Yes, I know you're busy with the SP right now. If this request needs to
> wait, that's fine.)

I'm no rsync expert, but...

rsync -avuC --delete -e ssh /opt/shibboleth-idp/metadata/dynamic/ webauth0:/opt/shibboleth-idp/metadata/dynamic

The interesting one is the flat file consumer:

#! /bin/sh

SRC=/home/shibboleth/etc/engineering-pantheon.txt
KEY=/home/shibboleth/certs/engineering-pantheon.pem
DEST=/home/shibboleth/idp/metadata/dynamic

CMD="/home/shibboleth/sbin/metagen.sh"

ORG="College of Engineering"
URL="http://engineering.osu.edu/"

while IFS= read -r ENTITY;
do
        if [ -z "$ENTITY" ] ; then
                continue;
        fi

        HASHED="$DEST/`echo -n "https://$ENTITY/simplesaml" | sha1sum | awk '{print $1}'`.xml"
        if [ -f $HASHED ] ; then
                echo "Duplicate filename generated for ($ENTITY)"
                continue;
        fi

        $CMD -2 -U -F -T SSP \
                -R eduPersonScopedAffiliation \
                -R OSUID \
                -R departmentNumber \
                -o "$ORG" -u "$URL" \
                -y "College of Engineering Pantheon Hosting" \
                -c $KEY \
                -h $ENTITY \
        > $HASHED 3> /dev/null

        xmllint --noout $HASHED
done < $SRC

The -R options I added to start triggering attribute release rules, but I'm just in my infancy with that. It looks like Unicon is suggesting some standard property names in the GUI to use for triggering attribute release, so once that settles we can get them documented as "reserved" attribute names by the project and cook up some defaults that will work with them.

-- Scott

--
To unsubscribe from this list send an email to [hidden email]