Building a composite attribute from sets of attributes

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Building a composite attribute from sets of attributes

Etienne Dysli Metref
Hi devs,

Here's another crazy SWITCH edu-ID extension that we would like to pull
off. We want to build a kind of composite attribute whose value is
composed of one or more sets of attributes, a bit like if you were
sending attributes for two different users without mixing them. If we
serialise this as JSON, the attribute value would look something like:

[
  {
    "eduPersonUniqueId": "[hidden email]",
    "eduPersonAffiliation": "staff",
    "idp": "https://uniA.example.org/idp/shibboleth",
    ...
  },
  {
    "eduPersonUniqueId": "[hidden email]",
    "eduPersonAffiliation": "student",
    "idp": "https://uniB.example.org/idp/shibboleth",
    ...
  }
]

I imagine this would need two separate attribute resolutions to fetch
the two sets for [hidden email] and [hidden email] without
mixing the attribute values together. We also want to filter those
attributes according to the policy set for IdP A and IdP B,
respectively, so two different filtering passes would be needed as well.

Is there a way to realise this with the current (3.4) IdP API?
I looked at classes in idp-attribute-filter-api and
idp-attribute-resolver-api modules and thought we'd need to implement
net.shibboleth.idp.attribute.filter.Matcher to do our special filtering,
but that's just a first guess.

This is starting to sound like OIDC's aggregated claims. If you plan to
implement that in the future, the same mechanism could be reused...

Cheers,
   Etienne


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Building a composite attribute from sets of attributes

Rod Widdowson
Without understanding what your desired endpoint is (and indeed the way that things are deployed in the field to any level of detail) all I can give is pure developers point of view.

If the idea is that you need an IdPAttribute whose values are IdPAtributes why not start with a class that implements IdPAttributeValue which does this?

You'd then need to write a Resolver and some Encoders and of course the parsers for these, plus of course any special filtering you need, but it's all public interfaces.

I'd suggest you look at where XMLObjectAttributeValue occurs in the source since that gives you a good idea of the plug points that would need

To re-emphasise, I have no idea whether this is the best domain specific way to approach this.

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
In reply to this post by Etienne Dysli Metref
> I imagine this would need two separate attribute resolutions to fetch
> the two sets for [hidden email] and [hidden email] without
> mixing the attribute values together. We also want to filter those
> attributes according to the policy set for IdP A and IdP B,
> respectively, so two different filtering passes would be needed as well.

The resolver and filter are abstracted significantly from all of that by using contexts that pass in information about the the user, the issuer/recipient, etc. When people write scripts that punch out of that layer into other pieces of the request, they're breaking that abstraction and sometimes things become SAML specific but the core behavior isn't.

> Is there a way to realise this with the current (3.4) IdP API?

It's not a function of the APIs we have not being "enough", it's a matter of the orchestration of those APIs into profile actions to implement flows, and your flow wouldn't be anything we have implemented now.
 
Using our action bean objects to wire up new flows isn't really supported because all the objects are implementation classes (e.g. the actions that run the resolver and filter now). It's possible to do it and people do it all the time but they're going to find that it all breaks in V4 since all those classes will undergo all sorts of small changes internally since they were for our use.

Formally, you'd have to clone all that code where it's needed and clone all the touch points that aren't API. It's not meant to be a capability we have. I'm considering looking at moving some of the action beans into the API so that it would become more practical to author complex flows like this without doing all the copying or risking breakage.

One of the main requirements is also to create and stash context objects in the tree in various ways, or to wire up the actions to go looking for particular contexts in different places so that there's no contamination.

The actions that do resolver and filter execution now are in idp-profile-impl.

What to do after all that would depend on what the end result was supposed to be and how it would get used. I doubt this is about SAML, so something else would be in the mix.

If the data here is really coming in as part of authentication back to different systems, that's also a different use case that I've been working on extensively for the next release.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
In reply to this post by Rod Widdowson
[Getting back to this after a week away from work...]

On 25/07/2019 17.23, Rod Widdowson wrote:
> Without understanding what your desired endpoint is (and indeed the
> way that things are deployed in the field to any level of detail) all
> I can give is pure developers point of view.

I'll try to quickly introduce our concept of "affiliation" then. What we
call by that name is the set of SAML attributes that a university's IdP
returns about a user. In SWITCH edu-ID, we capture that set, store it,
and associate it to that same person's edu-ID account.

Today, our SWITCH edu-ID IdP is capable of releasing either SAML
attributes coming from the edu-ID account -- the "private identity" --
*or* attributes coming from *one* of the user's affiliations, at the
user's choice. What we want to do now is send, in one SAML assertion,
attributes from both the private identity and all affiliations in a way
that keeps these different attribute sets separate. In order to keep the
structure, we want to send all affiliations as JSON-structured data in
one SAML attribute.

For more context around this, have a look at our public architecture
introduction [1] or at my TNC19 presentation [2] where I explain the
data challenges and our solutions.
> If the idea is that you need an IdPAttribute whose values are
> IdPAtributes why not start with a class that implements
> IdPAttributeValue which does this?

Yes, that would be a good start. In our case, that would be
`IdPAttributeValue<Set<IdPAttribute>>`.

> You'd then need to write a Resolver and some Encoders and of course
> the parsers for these, plus of course any special filtering you need,
> but it's all public interfaces.

For resolution, should we then implement a whole
n.s.i.attribute.resolver.AttributeResolver?

Couldn't we instead get by with a custom
n.s.i.attribute.resolver.dc.MappingStrategy<SearchResult>
(n.s.i.attribute.resolver.dc.ldap.impl.SearchResultMappingStrategy being
out of bounds) that would map each search result to an IdPAttribute
containing a Set<IdPAttribute>, plus our own
n.s.i.attribute.resolver.AttributeDefinition (in case
n.s.i.attribute.resolver.ad.impl.SimpleAttributeDefinition merges too much)?

  Etienne

[1] https://www.switch.ch/edu-id/organisations/architecture/
[2] https://tnc19.geant.org/sessions/#s54 (slides and video)


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
> In order to keep the structure, we want to send all affiliations as JSON-structured data in
> one SAML attribute.

I really wouldn't.

I think the essential issue is: who cares about this distinction? My guess is nobody will; the history of this sort of thing is that we think apps care about lots of things but apps care about nothing most of the time. So assuming they care and doing lots of work to let them care is usually premature.

if you have specific knowledge of them caring, then we can talk about it.

Note that scoped attributes were designed to already support this anyway.

In the meantime, there's a SAML extension defined for this use case called "OriginalIssuer". It's an XML attribute that can be added to <saml:Attribute> that carries the identity of a upstream issuer that's distinct from the assertion issuer.

Nothing supports this today because of point #1 (nothing cares). But it's relatively easy for us to add support for it if it's needed. I had already planned on probably looking at it for V4 beause of the proxying work. We just have to capture it in IdPAttribute, obviously.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
On 06/08/2019 14.22, Cantor, Scott wrote:
> I think the essential issue is: who cares about this distinction? My
> guess is nobody will; the history of this sort of thing is that we
> think apps care about lots of things but apps care about nothing most
> of the time. So assuming they care and doing lots of work to let them
> care is usually premature.

I'm with you on this one. I have yet to see a written justification from
the app side and the only answer so far is "because the project manager
said so"... which I've kicked back into the discussion arena today. ;)

We only have three services that really need some affiliation data and
these are satisfied by sending them all eduPersonScopedAffiliation
values that we have. All three are operated by SWITCH and the one we are
working on now would be the first external service.

> In the meantime, there's a SAML extension defined for this use case
> called "OriginalIssuer". It's an XML attribute that can be added to
> <saml:Attribute> that carries the identity of a upstream issuer
> that's distinct from the assertion issuer.

Oh that's interesting! and it goes where we want to: group attributes
and their values by affiliation in a way that preserves their origin
information.

There is, however, one case where this falls short: when a university
issues separate accounts for a person that is both student and staff.
Here attributes from both accounts come from the same IdP so their
association with the staff or student role is lost. For example,
eduPersonUniqueId or email values from the student account and the staff
account would be mixed together. We would like to find a solution that
keeps these separate.

Do you have time for a call to discuss this? I'm not sure I managed to
correctly explain what we are doing and want to achieve in writing.

  Etienne


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
On 8/6/19, 10:57 AM, "dev on behalf of Etienne Dysli Metref" <[hidden email] on behalf of [hidden email]> wrote:

> There is, however, one case where this falls short: when a university
> issues separate accounts for a person that is both student and staff.

I hate few things as much as I hate that practice.

> Here attributes from both accounts come from the same IdP so their
> association with the staff or student role is lost. For example,
> eduPersonUniqueId or email values from the student account and the staff
> account would be mixed together. We would like to find a solution that
> keeps these separate.

I'm not sure I would argue that those attributes should really be mixed; the whole point of that dual account strategy is to segregate access and not lump the information together in one session. Nothing you want to notice that distinction will notice it and you're right back where you started.

> Do you have time for a call to discuss this? I'm not sure I managed to
> correctly explain what we are doing and want to achieve in writing.

I think I get it, but if you want time on a dev call to talk about it, that's fine (next one is Aug 16th, 10-12 EDT).

Either way, if you really want JSON, just deal with it as a string value. Build a scripted attribute that pulls in everything else and cooks up the JSON you want to pass out as a single string value.
 
-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
On 06/08/2019 17.06, Cantor, Scott wrote:
>> There is, however, one case where this falls short: when a university
>> issues separate accounts for a person that is both student and staff.
>
> I hate few things as much as I hate that practice.

We should have banned that practice 15-20 years ago when starting
SWITCHaai. We now hope to eliminate it with SWITCH edu-ID.

The SWITCH edu-ID aims at bringing a user-centric, i.e. single identity
to the individuals. So, this is a nice concept, but it hits reality
soon, mostly at the services which today most often assume *a specific
role or affiliation* when the user logs in. As a consequence, the edu-ID
has to deal with the different roles or affiliations of a single person.
One of the tools we developed for that situation is the affiliation
chooser: just before the attribute consent step, the user chooses one
affiliation, i.e. specifies the set of attributes to present to the
service, out of the stored affiliations they have (usually not that many).

Now that we have properly segregated the information about different
affiliations in our user directory on the IdP, we noticed that there are
actually services (!) that would want to profit from *all* that data at
once. They would be handed over all information about all affiliations,
and then make some sense out of it like, e.g.:

- a nationwide library system that replaces 500+ single library systems,
and that needs to know of individuals who are member of one or more of
those, with all their respective attributes properly segregated and not
lumped together;

- commercial services that want to support end users in finding a
suitable affiliation allowing them to use the service, or some part of
the service. Or the service defines usage quota based on the contracted
organisations, and the user wants to use the sum of allowed quota for
all organisations they are a member of.

BTW we are talking about the following affiliation-specific attributes
in particular:
- eduPersonScopedAffiliation
- mail (we've got values that stem from the organisations)
- swissEduPersonUniqueID (as it is issued by the organisation)
- later: eduPersonEntitlements and maybe group information

One can -- up to a certain point -- let the user choose the values.
However, this becomes difficult with the swissEduPersonUniqueID when the
user has two roles at the same organization, which would look too
similar. The more attributes a user has to pick, the less user friendly
and the more error-prone the whole process gets.

> I'm not sure I would argue that those attributes should really be
> mixed; the whole point of that dual account strategy is to segregate
> access and not lump the information together in one session. Nothing
> you want to notice that distinction will notice it and you're right
> back where you started.
That's what I meant: attributes from dual accounts should not be lumped
together, which is what OriginalIssuer would do.

>> Do you have time for a call to discuss this? I'm not sure I managed to
>> correctly explain what we are doing and want to achieve in writing.
>
> I think I get it, but if you want time on a dev call to talk about
> it, that's fine (next one is Aug 16th, 10-12 EDT).
Great, thank you! We'll be there on the next call. We should stress
again that we're not expecting quick results here. For the time being
we'll be happy to see that our problem is understood by others.

Cheers,
  Etienne


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
In reply to this post by Cantor, Scott E.
On 06/08/2019 17.06, Cantor, Scott wrote:
> Either way, if you really want JSON, just deal with it as a string
> value. Build a scripted attribute that pulls in everything else and
> cooks up the JSON you want to pass out as a single string value.

We have an internal attribute like this already that builds a list of
affiliations for the affiliation chooser intercept flow to display. At
first, we used a Template attribute definition, but ended up building a
"soft template" in a Scripted attribute to fail more gracefully when
source attributes don't have the same number of values. This is terribly
ugly because it relies on an ordering we have no guarantees over. The
script does this:

  for (i = 0; i < swissEduPersonUniqueID.getValues().size(); i++) {
    var valueParts = [swissEduPersonUniqueID.getValues().get(i),
      swissEduIDAttributeProviderIdentifier.getValues().get(i),
      swissEduPersonHomeOrganization.getValues().get(i),
      swissEduPersonHomeOrganizationType.getValues().get(i)];
    currentAffiliations.addValue(valueParts.join("!"));
  }

Values for the four used attributes are assembled by the resolver from a
LDAP subtree query under ou=affiliations with filter (swissEduID=XXXX).
The corresponding DataConnector also has maxResultSize="0". Where the
LDAP data looks like:

swissEduPersonUniqueID=[hidden email],ou=affiliations,...
  swissEduID: XXXX
  swissEduPersonUniqueID: [hidden email]
  swissEduIDAttributeProviderIdentifier: <uniA's IdP>
  swissEduPersonHomeOrganization: uniA.example.org
  swissEduPersonHomeOrganizationType: university

swissEduPersonUniqueID=[hidden email],ou=affiliations,...
  swissEduID: XXXX
  swissEduPersonUniqueID: [hidden email]
  swissEduIDAttributeProviderIdentifier: <uniB's IdP>
  swissEduPersonHomeOrganization: uniB.example.org
  swissEduPersonHomeOrganizationType: university

Correct me if I'm wrong, but the IdP's resolver groups all found
attribute values without any particular ordering guarantee. Moreover,
LDAP doesn't provide an ordering on returned attribute values. So this
happens to work by chance and we get:

[hidden email]!<uniA's IdP>!uniA.example.org!university
[hidden email]!<uniB's IdP>!uniB.example.org!university

but it could totally blow up on us and instead produce

[hidden email]!<uniB's IdP>!uniB.example.org!university
[hidden email]!<uniA's IdP>!uniA.example.org!university

any day... :O

  Etienne


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Building a composite attribute from sets of attributes

Rod Widdowson
> Correct me if I'm wrong, but the IdP's resolver groups all found
> attribute values without any particular ordering guarantee.

Absolutely not.  AttributeValues ordering within an Attribute is guaranteed, this is required to make templating make sense.  By extension this is why we have Null attribute value types.

 In V4 we have tightened things up to that the ordering is embedded in the API contract (which is a pompouse way of saying that I replaced Collection<Value> with list<Value>.

The ordering *does* break down after filtering.

> Moreover,
> LDAP doesn't provide an ordering on returned attribute values.

That’s 'unfortunate'...

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
In reply to this post by Etienne Dysli Metref
On 8/7/19, 9:16 AM, "dev on behalf of Etienne Dysli Metref" <[hidden email] on behalf of [hidden email]> wrote:

> Correct me if I'm wrong, but the IdP's resolver groups all found
> attribute values without any particular ordering guarantee.

That's LDAP, not the IdP. The IdP absolutely maintains order (it has to for SQL to work) so I would say this is not solveable with LDAP to start with if the data is spread across different attributes, but would be trivially solveable with a database instead.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
In reply to this post by Rod Widdowson
On 07/08/2019 15.18, Rod Widdowson wrote:
> Absolutely not.  AttributeValues ordering within an Attribute is
> guaranteed, this is required to make templating make sense.  By
> extension this is why we have Null attribute value types.

I happily stand corrected then. :)

Coming back to my fake LDAP data

swissEduPersonUniqueID=[hidden email],ou=affiliations,...
  swissEduID: XXXX
  swissEduPersonUniqueID: [hidden email]
  swissEduIDAttributeProviderIdentifier: <uniA's IdP>
  swissEduPersonHomeOrganization: uniA.example.org
  swissEduPersonHomeOrganizationType: university

swissEduPersonUniqueID=[hidden email],ou=affiliations,...
  swissEduID: XXXX
  swissEduPersonUniqueID: [hidden email]
  swissEduIDAttributeProviderIdentifier: <uniB's IdP>
  swissEduPersonHomeOrganization: uniB.example.org
  swissEduPersonHomeOrganizationType: university

Given those two results, the output of the DataConnector will always be:

swissEduPersonUniqueID: [hidden email], [hidden email]
swissEduIDAttributeProviderIdentifier: <uniA's IdP>, <uniB's IdP>
etc.


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
On 8/7/19, 10:27 AM, "dev on behalf of Etienne Dysli Metref" <[hidden email] on behalf of [hidden email]> wrote:

> Given those two results, the output of the DataConnector will always be:

If the LDAP entries coming back are separate entries, then yes, the individual values will line up as long as the number of values isn't different.

If they're one entry then the order is up to the LDAP server. I was thinking it was one entry, but I imagine not.

Your script may be ugly, but there's nothing less ugly you could do. Writing a dozen Java plugins to support something that will ultimately have to get turned into the same thing in the end isn't going to improve things.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
On 07/08/2019 16.33, Cantor, Scott wrote:
> If the LDAP entries coming back are separate entries, then yes, the
> individual values will line up as long as the number of values isn't
> different.
>
> If they're one entry then the order is up to the LDAP server. I was
> thinking it was one entry, but I imagine not.

Oh they are separate entries, sorry if that wasn't clear.

> Your script may be ugly, but there's nothing less ugly you could do.
> Writing a dozen Java plugins to support something that will
> ultimately have to get turned into the same thing in the end isn't
> going to improve things.

I'd prefer to have that script covered by unit tests, yet building a
test harness to execute that bit of JavaScript the same way the IdP does
is too complicated and brittle. That's why I favour Java, testing is
easy. However, the less code the better, so if we get the same effect
with much less JavaScript than Java I'll live with it...

  Etienne


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
On 8/7/19, 11:15 AM, "dev on behalf of Etienne Dysli Metref" <[hidden email] on behalf of [hidden email]> wrote:

> I'd prefer to have that script covered by unit tests, yet building a
> test harness to execute that bit of JavaScript the same way the IdP does
> is too complicated and brittle. That's why I favour Java, testing is
> easy. However, the less code the better, so if we get the same effect
> with much less JavaScript than Java I'll live with it...

I'm not opposed I guess to just putting together some kind of wacky JSON AttributeDefinition if we can make sense of the requirements for it, but I'm not sure it's got much general appeal.

We do have unit tests for scripted stuff, maybe we could help out there somehow.

Also, we could maybe come up with Java code to do part of the work, but leave it being run by the script.

What I was trying to say was that formalizing the concept of a new form of AttributeValue implies a great deal of plumbing all over the system.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Etienne Dysli Metref
In reply to this post by Cantor, Scott E.
On 07/08/2019 16.33, Cantor, Scott wrote:
> If the LDAP entries coming back are separate entries, then yes, the
> individual values will line up as long as the number of values isn't
> different.

So far my examples and my thinking only had single-valued attributes,
but then it breaks down for multi-valued attributes...

From a scripted attribute, is there a way to access something that looks
like the LDAP entries? I mean having the same structure, or is that
already gone at this point (processed by SearchResultMappingStrategy)?

  Etienne


--
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Rod Widdowson

> From a scripted attribute, is there a way to access something that looks
> like the LDAP entries? I mean having the same structure, or is that
> already gone at this point (processed by SearchResultMappingStrategy)?

It’s long gone by then...
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building a composite attribute from sets of attributes

Cantor, Scott E.
In reply to this post by Etienne Dysli Metref
On 8/8/19, 3:58 AM, "dev on behalf of Etienne Dysli Metref" <[hidden email] on behalf of [hidden email]> wrote:

> So far my examples and my thinking only had single-valued attributes,
> but then it breaks down for multi-valued attributes...

I think you have to control the LDAP structure more directly, or move off LDAP. If what you need is a single value, then storing the data in a delimited string as a single value is a better model than multiple values, even if that would otherwise be a bad thing. Every situation is unique. The data model should serve the need, not the other way around.
 
-- Scott


--
To unsubscribe from this list send an email to [hidden email]