Ubuntunet Connect 2017 - Service Development Clinic

events
devops
tdd
infra-as-code
workshop

(Bruce Becker) #1

@ChrisRohrer is organising a service development clinic for NRENS at the next Ubuntunet Connect conference in Addis (30-31 October). From initial discussions, this will be a bit of a mashup between DevOps Bootcamp format (previously held in Entebbe and Pretoria) and the e-Research Hackfest. Participants will be hacking their own services, and learning the tools of the trade as we go.

While we're still discussing the format and timetable, I wanted to bring one topic to the fore : testing.

I've spent the last few weeks investigating tools for running independent tests on infrastructure code, to have a better quality of deployment. Too often some provisioning code is written (say, a playbook and role), and it either misses or wrongly implements some aspect which is required down the line. In my case it was network security. I wanted to write tests independently of the deployment - with the hope that tests could be used as a concrete expression of an SLA or something. Like, if a site has a guideline on how to expose ports or usernames, or best practice benchmarks, or is monitored by Nessus or whatever.

This would better bridge the gap between Dev and Ops, and is something we really haven't taken into account yet, although we always professed to do so.

After taking a look around, the tools for testing are essentially :

These rely on ruby (Rspec) or python (pytest) frameworks, and can be mixed-and matched with various provisioners, testers, etc.

I think it would be nice to have a session where we go over the various options, to see what patterns are used, and then try to write specs for compliance with CIS benchmarks

If we're going to be using Ansible, there's a great new testing tool molecule which can be used to improve the dev cycle, where you init roles and tests at the same time, essentially implementing TDD for infrastructure.

@ChrisRohrer what do you think ?


(Chris Rohrer) #2

Heaps of good ideas in here, although I must admit that I have never heard of some of the testing tools you mention. Will definitely look into them.

I am still a bit unsure about the name of the workshop. 'Service Clinic' doesn't really say what we plan to do, does it? How about 'Service Deployment Clinic'? I consider this training to be more about deployment and operation that actual development work.

I have switched over from Ansible to Docker over the last few months. For many services I still have the Ansible playbooks lying around, but nowadays when I need to provision a new service, I usually take the time to set it up as a Docker Stack. The main advantage of Docker for me is the fact that it is deployed within seconds. Running Ansible from my local machine on the slow connection I am using is a hassle and takes ages.

This are some initial thoughts, let me have a look at all the tools mentioned in your post and come back with a better-informed response regarding Testing after that.


(Bruce Becker) #3

Hey @ChrisRohrer I took a look at the spreadsheet you sent a few moments ago. It looks like people are mostly interested in setting up eduroam and institutional repositories. I think this fits nicely into the themes of the conference actually - I think it would be a good idea to have a bit of a broader picture for the attendees, especially for the institutional repositories - how they fit into directories of archives, metadata harvesters, etc. This is important for the African Open Science Platform too.

We could give a recap of what we did in Sci-GaIA for the invenio deployment. If people are married to D-Space I'm going to have to do some homework, but if they just want an institutional repos, we have off-the-shelf stuff for them to use. The most important thing to do in this clinic however is to talk about the specifics of how to deploy and operate these things properly, right ?