GNOME SysAdmin Andrea Veri about GNOME Infrastructure

  I was really young when I first started looking into Open Source projects. It was around 2005, I was just fifteen years old and my very first distribution was Fedora. During that years a Linux magazine and a party at a friend’s house made me discover Ubuntu. Everything looked great, especially the community that was driving the project. I really wanted to understand how the whole process worked, how the OS was being built and what was the community role in all that. I decided to join the Ubuntu’s italian LoCo Team, became an Ubuntu member and after a few months an Ubuntu Developer.

As you may know Ubuntu is built upon Debian, and while forwarding my Ubuntu patches to the Debian Developers I started being involved in Debian as well. On the fifth of January 2010 I had the great pleasure and honor to join the Debian Developer ranks. I’ve always been an heavy GNOME user and it was time for me to start giving back to the project in some way and that’s where my collaboration with the Debian GNOME Team started. I did several packages and heavily contributed to the GNOME 2.28.0 release on Debian Sid. I met awesome people and developers, from Josseline Mouette, Jordi Mallach, Sjoerd Simons, Emilio Pozuelo Monfort and I really feel they’ve been an inspiring force for me to learn new things and understand how the whole processes worked out.

My Debian contributions to GNOME packages helped me joining the GNOME Foundation back in November 2009. During that period the Membership Committee (the committee that reviews and eventually accepts new Foundation members) and the GNOME Accounts team (the team that manages GNOME’s LDAP) needed help. I threw my hat in and started cleaning the dozens of outstanding tickets on the queues, reviewed the outdated wiki pages, rebuilt how the whole procedures were working out and simplified a bit the process itself. During the same period I started my involvement on the Fedora Project as a packager and SysAdmin. Unfortunately I don’t have much time to cover my Fedora activities completely but given the fact some of the GNOME servers are hosted on Red Hat’s datacenters, I’m daily in touch with the Fedora SysAdmins. I’m a lucky boy after all, working with experienced administrators like Kevin Fenzi, Stephen Smoogen and Seth Vidal is amazing, you really never stop learning new things from them and their enthusiasm has been a driving force for me during the past months.

Around one year ago I joined the GNOME SysAdmin team, I’m sure you’ve heard my name during the past outages :-)

  The GNOME SysAdmin team is currently composed by five people. Apart from me there’s Jeff Schroeder, Olav Vitters, Owen Taylor and Sriram Ramkrishna. Each of us has the responsability to make sure all the GNOME services and processes are running smoothly. We are also in charge of deploying new services (like Owncloud for the GNOME 3.8 release), evaluating proposals from the community about adding new resources or switching existing resources to other softwares. (i.e Gitorious)

  My past experiences helped me a lot in feeling at home with all the distributions we run on our servers. Most of the machines run RHEL and Ubuntu, and switching from Deb to RPM based systems usually helps me a lot in not forgetting all the syntaxes and peculiarities of each system. Honestly speaking when you understand the basics of setting up a server it won’t matter which server distribution are you actually using. What I always try to do is installing new services on standard directories so that finding out where a specific website lives is a matter of seconds on either Ubuntu or RHEL systems.

There is no fixed amount of time for my daily contributions, but when something goes wrong or a service migration has to happen I can even spend 6-7 hours working on it. A lot of people depend on my work and going to sleep when something is broken and people are waiting for your action makes me not sleeping at all.

  Pretty much all the GNOME servers are hosted by Red Hat, Canonical and OSUOSL but owned by the GNOME Foundation itself thanks to either personal or companies donations. Receiving new hardware requires a lot of time and proposals and we’re currently in the middle of that for having a few old machines retired and replaced by some fresh new hardware. I’m not sure how long it will take but I’ll cooperate with the relevant parties to have this sorted out as soon as possible.

Olav Vitters (bkor) just started testing out the latest Bugzilla release on a Virtual Machine but while there is a plan to upgrade our Bugzilla istance sooner or later there’s no fixed time for that to happen. The main issue is importing all the GNOME-specific extensions to the new release and that unfortunately requires a lot of time.

  I’m personally not aware of any big Open Source distribution using GitHub or Google Groups, especially for the fact the software ran on these services is not Open Source at all. Ubuntu, Debian, Fedora have their own infrastructure, all their applications are strictly based upon Open Source software. GNOME does the same.

The idea of having your self-hosted infrastructure is making contributors life easier. You can configure the software as you wish, you can make it act as you want and when you want. Google and GitHub gives you a product which has to be used as it is, if you don’t like a single thing of it there’s nothing you can do apart sending a proposal request and hoping it’ll be eventually evaluated in the next thousand years. I personally think we can adapt the idea behind the Open Source movement to how we manage our servers and applications. We manage our infrastructure, our applications, our services, our scripts and we are free to adapt them to suit our needs as much as we can.

There’s always space for improvements and I am still waiting for a concrete community proposal (what are the benefits, what could be improved in our Git infrastructure, etc.) for eventually switching git.gnome.org to Gitorius, which is Open Source software unlike GitHub. Apart from that what we could really improve is how our services are connected to the outside world, it would be great to have one single user and password for all the services hosted on the GNOME infrastructure, it would be also great to have OpenID logins enabled everywhere but that requires more time than the one our life permits us to spend on our FOSS contributions.

  There’s currently a discussion about how OwnCloud should be integrated within the GNOME infrastructure, specifically is the service meant for Foundation members or for the “big public”? I feel we don’t have the needed resources to handle hundreds of thousands users at this moment of time but ideally we could have our needs covered by outsourcing the service to an external provider, like Dropbox does with Amazon.

GNOME cares a lot about its user’s privacy and when you run a service on your infrastructure that receives a lot of sensitive data you need to make sure you have a strong and defined Privacy Policy and Usage Agreements. As for myself I wouldn’t adopt Dropbox’s policy specifically where they grant law enforcement agencies to access your files under certain conditions. (as per “Compliance with Laws and Law Enforcement Requests; Protection of Dropbox’s Rights”, paragraph 3 at https://www.dropbox.com/dmca#privacy)

What I would do as a GNOME lawyer is opting for a Privacy Policy like the one provided by the SpiderOak cloud solution, which states that data stored on their Cloud is encrypted and inaccessible without user’s key, which is stored locally on user’s computers. That actually means that while Dropbox’s employees can do whatever they want with your data, SpiderOak’s ones can’t since they’re not able to access any of the information stored on their servers without the keypair stored on the user’s machine. This is just my personal opinion though.

  Luckily we did have one intrusion so far during the past year. The target machine was the one hosting www.gnome.org and the cause was an outdated version of Plone standing on the server without anyone knowing of it. GNOME has a lot of contributors and many services were set up for testing or development purposes especially when the new GNOME website was being built. The possible choices were Plone or WordPress and the SysAdmin team did setup the needed infrastructure for the website team to test all the possible choices. When the decision was made and WordPress was choosen as the desired platform, the Plone istance was left there unmaintained. After that happening we decided to shutdown some useless and outdated services and rebuilt the machine to be extremely sure nothing was touched except the files / folders owned by the Plone user.

  We had more than 100 new developers enabled to access GNOME’s Git repositories, we had a completely new service called Extensions, which I’m sure many of you use daily. Also Colin Walters is currently involved on what will be the substitute of JHBuild, called OSTree. Our current resources are covering all our needs but many machines are getting old and are going to be retired really soon. There’s also a plan to set up a complete DevOPs environment with the the ability to allocate and deallocate virtual machines in an automated fashion, so moving from virtualization to a Cloud solution is definitely one of the things we’ll be looking forward to deploy on our infrastructure.

It also has to be said that a lot of traffic happens on the relevant distributions resources and not on the GNOME infrastructure itself. Most of the time users join a Fedora or an Ubuntu forum to ask a GNOME-related question, thus the GNOME-hosted-resources are not heavily used by GNOME users themselves but by developers and contributors instead.

  I’m going to get a degree in law really soon and after that I plan to start studying Web Technologies. The chinese philosopher Confucio said “Choose a job you love, and you won’t have to work a single day of your life”, I dream about working in something I really love and care in my life and informatics and technology will be my very first choice.