Package mirroring / Jisc

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Package mirroring / Jisc

Cantor, Scott E.
Jisc, which is the consortium operator and one of the founding board members, has offered to work this year on moving the packages into their Azure cloud, which will mirror things in a few locations around the world in one step. I don't have an ETA on that happening.

We're still willing to rsync or lftp the files to any members or anybody willing to offer open access, and there are a couple of offers for that, nothing in place yet.

Separately, just FYI the data center where our server lives has been having network issues all morning. [1]

-- Scott

[1] http://status.iweb.com/incidents/110272
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Package mirroring / Jisc

Chris Andre
Hi Scott,

Hope all is well just picking this up and as Consortium Members we would love to host the shibboleth files.

I was wondering if you could give me some statistics on the current usage (i.e bandwidth requirements, space requirements etc) and I am sure we can get some server space setup as a mirror.

Thanks

Kindest Regards

Chris

On 14 March 2018 at 14:46, Cantor, Scott <[hidden email]> wrote:
Jisc, which is the consortium operator and one of the founding board members, has offered to work this year on moving the packages into their Azure cloud, which will mirror things in a few locations around the world in one step. I don't have an ETA on that happening.

We're still willing to rsync or lftp the files to any members or anybody willing to offer open access, and there are a couple of offers for that, nothing in place yet.

Separately, just FYI the data center where our server lives has been having network issues all morning. [1]

-- Scott

[1] http://status.iweb.com/incidents/110272
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]



--

-- 

Chris Andre

CTO, Overt Software Solutions Ltd

For support queries please email [hidden email] to ensure that your query is sent to all available engineers.

Cloud Hosting / Dedicated Servers | Shibboleth | Moodle | Cloud backups & file sharing


T: 0330 2000 141 | E: [hidden email] | www.overtsoftware.com



--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Package mirroring / Jisc

Cantor, Scott E.
> I was wondering if you could give me some statistics on the current usage (i.e
> bandwidth requirements, space requirements etc) and I am sure we can get
> some server space setup as a mirror.

We don't host them so we have no statistics other than size.

I posted all the relevant details at [1] excepting that we're happy to use lftp instead of rsync, since it's generally easier for the host if you're familiar with it.

-- Scott

[1] http://shibboleth.net/downloads/service-provider/2.6.1/RPMS/
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Package mirroring / Jisc

Peter Schober
* Cantor, Scott <[hidden email]> [2018-03-15 15:43]:
> I posted all the relevant details at [1] excepting that we're happy
> to use lftp instead of rsync, since it's generally easier for the
> host if you're familiar with it.

Script (and Ansible role) to securely set up a server to act as push
mirror target forthcoming.
-peter
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Package mirroring / Jisc

Rhys Smith-2
In reply to this post by Cantor, Scott E.
On 14 Mar 2018, at 14:46, Cantor, Scott <[hidden email]> wrote:
>
> Jisc, which is the consortium operator and one of the founding board members, has offered to work this year on moving the packages into their Azure cloud, which will mirror things in a few locations around the world in one step. I don't have an ETA on that happening.

Should be some time over the summer, I think.

Rhys.
--
Dr Rhys Smith
Chief Technical Architect, Trust & Identity
Jisc

T: +44 (0) 1235 822145
M: +44 (0) 7968 087821
Skype: rhys-smith
GPG: 0x4638C985
Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG

jisc.ac.uk

Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.


--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

ansible script for setting up a Shib RPM mirror (was: Re: Package mirroring / Jisc)

Peter Schober
In reply to this post by Peter Schober
* Peter Schober <[hidden email]> [2018-03-15 20:12]:
> * Cantor, Scott <[hidden email]> [2018-03-15 15:43]:
> > I posted all the relevant details at [1] excepting that we're
> > happy to use lftp instead of rsync, since it's generally easier
> > for the host if you're familiar with it.
>
> Script (and Ansible role) to securely set up a server to act as push
> mirror target forthcoming.

Using this Ansible role anyone should be able to easily and securely
configure a machine as a Shibboleth package mirror:

  https://github.com/peter-/shib-rpm-mirror
  https://galaxy.ansible.com/peter-/shib-rpm-mirror/

If things are unclear from the existing documentation included above I
can certainly expanded on that (or create a page on the Shibboleth wiki).

I've successfully tested this on clean/minimal installs of:

* Debian 9 ("Stretch") and 8 ("Jessie")
* Ubuntu 16.04 LTS ("Xenial") and 14.04 LTS ("Trusty")
* CentOS 7 and CentOS 6

The README has additional disclaimers/instructions for some those, though.
Personally I'd be fine with questions/issues reported here or via
github issues.

Hope this motivates more mirror sites to be contributed (esp from
consortium members, or maybe from those not able to join yet) to get
rid of those OBS errors for good.

-peter

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: ansible script for setting up a Shib RPM mirror (was: Re: Package mirroring / Jisc)

Cantor, Scott E.
Thanks to Peter for working with me on this. We now have a relatively workable system that will be relying on a centrally hosted mirror list that I've put together, so I've updated the repo generation script [1] to produce an indirected reference to that mirror set, which currently only has the Austrian mirror in it.

I decided to omit the opensuse mirror for the simple reason that it just doesn't work, but if this is a problem for Peter's server before we get more mirrors online, I'll adjust it.

With this change the repo files look like this (platform aside):

[shibboleth]
name=Shibboleth (CentOS_7)
type=rpm-md
mirrorlist=https://shibboleth.net/cgi-bin/mirrorlist.cgi/CentOS_7
gpgcheck=1
gpgkey=https://downloadcontent.opensuse.org/repositories/security:/shibboleth/CentOS_7/repodata/repomd.xml.key
enabled=1

This should be permanent going forward and we'll add mirrors as they come online, so it should work as we'd all expect. I will update the docs in the wiki as time permits.

-- Scott

[1] https://shibboleth.net/downloads/service-provider/RPMS/

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: ansible script for setting up a Shib RPM mirror (was: Re: Package mirroring / Jisc)

Cantor, Scott E.
> This should be permanent going forward and we'll add mirrors as they come
> online, so it should work as we'd all expect. I will update the docs in the wiki
> as time permits.

An early tester found a missing package from the new mirror and that's been corrected along with the underlying cause. We'll also be clarifying, please just file bugs at https://issues.shibboleth.net if any problems with a mirror occur, so we can triage and reach out as needed, no need to pester the mirror owner as they probably aren't at fault (and unlike OBS, we actually care about the mirrors working, so we'll deal with it).

-- Scott

--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Package mirroring / Jisc

Peter Schober
In reply to this post by Chris Andre
Chris,

* Chris Andre <[hidden email]> [2018-03-15 15:32]:
> as Consortium Members we would love to host the shibboleth files.

That's great, you can arrange details at <[hidden email]>.

> I was wondering if you could give me some statistics on the current
> usage (i.e bandwidth requirements, space requirements etc) and I am
> sure we can get some server space setup as a mirror.

Bandwidth: The first 10 weeks of running such a mirror have caused
~860 MiB of incoming traffic (likely including minor package uploads)
and less than 13 GiB outgoing, according to vnstat.
Each new mirror would reduce that even further, of course.

Disk size: The current YUM archive is ~11 GB on disk. So nothing to
lose sleep over, either.

-peter
--
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to [hidden email]