Modular Installation in the IdP

Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Modular Installation in the IdP

Rod Widdowson
I've been kicking some ideas around about how we approach adding new functionality that not everyone needs without increasing the
size of the monolithic IdP install.

I have come up with a loose proposal which I'd like to get some discussion of.

I've put it up in the DEV part of the wiki [1].  We can talk about it here or on Friday.

I'll take feedback about anything, but in particular I'd like help in the area of signing.

Thanks

        Rod
[1] https://wiki.shibboleth.net/confluence/display/DEV/ModuleInstallation

--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Modular Installation in the IdP

Rod Widdowson
To follow up on my mail last week

> I've been kicking some ideas around about how we approach adding new functionality that not everyone needs without increasing
> the size of the monolithic IdP install.

        1) After discussions last Friday we decided to call this "plug ins" so as to try to separate this from the java "module"
thing.  Hence the document has been renamed [1]
        2) I have tried to capture the feedback I have received so far in the document.
        3) I have added a couple of skeletal sections at the end to do with testing and project/process issues.

As before comments come here or go into the comments for the page.

Thanks all

        Rod

[1] https://wiki.shibboleth.net/confluence/display/DEV/Plug-in+Installation


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Modular Installation in the IdP

Cris Rockwell
An OSGi version of the OpenSAML modules could achieve the same effect.

Cris Rockwell


On Apr 22, 2020, at 8:53 AM, Rod Widdowson <[hidden email]> wrote:

To follow up on my mail last week

I've been kicking some ideas around about how we approach adding new functionality that not everyone needs without increasing
the size of the monolithic IdP install.

1) After discussions last Friday we decided to call this "plug ins" so as to try to separate this from the java "module"
thing.  Hence the document has been renamed [1]
2) I have tried to capture the feedback I have received so far in the document.
3) I have added a couple of skeletal sections at the end to do with testing and project/process issues.

As before comments come here or go into the comments for the page.

Thanks all

Rod

[1] https://wiki.shibboleth.net/confluence/display/DEV/Plug-in+Installation


--
To unsubscribe from this list send an email to [hidden email]


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Modular Installation in the IdP

Cantor, Scott E.
On 4/22/20, 11:00 AM, "dev on behalf of Cris Rockwell" <[hidden email] on behalf of [hidden email]> wrote:

> An OSGi version of the OpenSAML modules could achieve the same effect.

The OpenSAML modules (and they're Maven modules for arbitary code organization reasons, to be clear, not Java modules)  are not the plugins we're talking about.

The IdP is based on Spring and Spring Web Flow and plugins to the IdP have to supply configuration and bean files, along with jars, into a variety of Spring contexts throughout the existing architecture including new webflows, which we don't control the Spring context management for.

There is no isolation of classes to specific plugins possible with the current design and that's not a goal.

If we were starting from scratch, OSGI might be a consideration, but it's too late for that. It may be suitable to some use cases we hit that don't involve flows, but the need for a plugin management approach and securing the installation and upgrade process would be a need either way and could accommodate that.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Modular Installation in the IdP

Etienne Dysli Metref
In reply to this post by Rod Widdowson
On 13/04/2020 14.32, Rod Widdowson wrote:
> I've been kicking some ideas around about how we approach adding new
> functionality that not everyone needs without increasing the size of
> the monolithic IdP install.

As an IdP extension developer, I appreciate this effort toward more
modularity. :) What I still don't like about it, is the WAR rebuild step
which has to happen outside of my usual Java development workflow and
with a different tool. I've tried to work around this with Maven WAR
overlays, which I don't think will work with this new plug-in architecture.

What if deployers didn't need to build an IdP WAR file? This brings me
to another idea I've been kicking around and I'll summarise it with "own
your servlet container". The IdP could ship it's own servlet container
by embedding Jetty or Tomcat akin to what Spring Boot does, but without
the complicated JAR-in-JAR packaging. Then starting the IdP becomes a
matter of invoking Java with a classpath containing all "system" classes
and plug-ins (coming from different directories). This greatly
simplifies installation because deployers no longer have to maintain
Jetty or Tomcat themselves (likely for the sole purpose of running the
IdP) and there is just one command to start the IdP `java -cp ...
MainIdPClass` which is easily integrated with systemd. This is also
orthogonal to the plug-in installation/update machinery.

Of course, this makes you responsible for maintaining the embedded
servlet container, but on the other hand, it reduces the number of
different supported servlet containers to just one: the one you choose
to embed.

Given that Tomcat has been removed from RHEL 8 in favour of RedHat's own
JBoss product [1], it brings it on a equal footing with Jetty: neither
are packaged by RPM-based distributions for easy security updates.
People using these distributions are left manually upgrading their
servlet containers. I think it would help them if it were embedded in
the IdP so they don't have to maintain it. By the way, Tomcat's removal
from RHEL negates the biggest argument we (SWITCH) had to recommend
using it over Jetty for ease of maintenance. Meanwhile, users of Debian
and Ubuntu still enjoy packaged Tomcat and Jetty. :)

Embedding your own web server is also a tendency I can observe among the
web applications I operate as a developer: Jenkins, SonarQube, Sonatype
Nexus. They all run their own embedded servlet containers. It also makes
running these in [Docker] containers much easier.

What do you guys think?

  Etienne

[1]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/considerations_in_adopting_rhel_8/dynamic-programming-languages-web-servers-database-servers_considerations-in-adopting-rhel-8#tomcat-removal_configuring-the-unversioned-python
--
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Modular Installation in the IdP

Cantor, Scott E.
On 5/15/20, 6:08 AM, "dev on behalf of Etienne Dysli Metref" <[hidden email] on behalf of [hidden email]> wrote:

> As an IdP extension developer, I appreciate this effort toward more
> modularity. :) What I still don't like about it, is the WAR rebuild step
> which has to happen outside of my usual Java development workflow and
> with a different tool.

I don't undertand why this is a problem *for deployment*, but I'm sure we could work on some kind of tool that's better or an alternative. But I really would have to think that anything fully scriptable should be relatively interchangeable as an approach.

We also might want to get clearer about what you mean by "development workflow". We obviously develop and we do not build warfiles to rrun and debug the code locally. That doesn't obviate the need to be able to debug into running containers, which is a different problem, but in terms of *development* workflow, we don't do that, though it's not trivial obviously to avoid it.

I sort of assumed this was more about *deployment* workflow. Maybe we need to step back and understand the specific problem of having a warfile, which I don't think is this particular plugin discussion either.

> This brings me to another idea I've been kicking around and I'll summarise it with "own
> your servlet container". The IdP could ship it's own servlet container
> by embedding Jetty or Tomcat akin to what Spring Boot does, but without
> the complicated JAR-in-JAR packaging.

Well, I think you acknowledged this is a very orthogonal discussion, but if you want to have it...

Embedding a container only "solves" things by requiring proxying from Apache or some other front end. With us owning the servlet container, customizing it adequately for real world use becomes intractable.

If you really meant "no WAR file" in the "run a web server inside our code" sense, that's even more out of bounds IMHO as a *production* deployment approach without assuming a proxy. And I rather imagine that's what you're assuming?

It's a debateable point worthy of discussing as to whether it's better for us in the end to ship a non-viable web story that has to be instantly proxied by Apache or a load balancer for production use, but gives us the ability to not actually lay out the code in servlet container form.

I don't personally buy that it's simpler in ways that should really matter but I can accept that others may feel it is.

> This greatly simplifies installation because deployers no longer have to maintain
> Jetty or Tomcat themselves (likely for the sole purpose of running the IdP)

That's just not the case IMHO. They have to maintain *a* web front-end regardless, and now they have two to run (adding overhead and brittleness and security exposures, as all proxying does). Obviously if you know the other web server better, that's attractive, but the added work of deploying a container that isn't doing any of the real front-end work is just not that substantial either.

> Embedding your own web server is also a tendency I can observe among the
> web applications I operate as a developer: Jenkins, SonarQube, Sonatype
> Nexus. They all run their own embedded servlet containers. It also makes
> running these in [Docker] containers much easier.

All of them are either much more trivial in their web requirements, security disasters I would never expose to the Internet, or are simply punting on the problem. We are simply honest enough to admit that packaging a web server doesn't give you a web server, it just gives you an open HTTP port you have to proxy with a real web server (and/or load balancer, etc.)

And even so I'm still not opposed to it, but maybe the real question is, if you can't use the existing Internet2 packaging that uses Docker, doesn't that suggest that any single deployment package of this sort is probably unattainable? Everybody thinks they have the answer and that answer seems to be unique to every person.

My suggestion is that if this is the direction people want, we need people with the necessary skill sets to join the project to make it happen. We have resources to fund work if it's valued by the members. But we don't have the right people to do that work.

What I would say right now is that we don't intend or desire to make things worse with the plugin proposal. All we're trying to do is solve right now is the module management problem so functionality can be maintained more independently without making the installation or upgrade process any harder. In a perfect world maybe we'd have started with OSGi but we're just too far past that point.

I don't think any of that work is going to prevent any future directions that are taken related to the servlet container or use of the on-disk WAR assumption. If it does, I think we need to prevent that.

-- Scott


--
To unsubscribe from this list send an email to [hidden email]