Appliance testing framework

[JedMeister]

It'd be great if we had an appliance testing framework. This could be used to create a raft of test we could do on appliances.

@qrntz started wortk on a framework that leverages PhantonJS.

@Dude4Linux suggested that perhaps Selenium might be a better option. @qrntz had some concerns about that having a lot of x11 dependencies but it seems that the Selenium Server is desinged to run the tests headlessly (see here). Actually there is a 3rd party PhantonJS "driver" avaialble for Selenium (called GhostDriver).

Here are some other links that may be of interest:
http://docs.seleniumhq.org/download/
https://github.com/angular/protractor/blob/master/docs/server-setup.md
https://www.npmjs.com/package/selenium-standalone
http://nightwatchjs.org/

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[SiKing]

Just noticed this today.
Selenium is a browser-app automation framework, so you would only be able to automate browser stuff. Not sure if that is what you are after.
GhostDriver gives you Java bindings for PhantomJS. This is good, assuming you want the framework written in Java. Also worth considering, from the GhostDriver page: "I have been away from this project for 2+ years. ..."
Anyway, if this is the direction you would like to go, I can definitely help out.

[JedMeister]

TBH @SiKing I have no idea about the intricate details of this stuff. It was just a dump of some info that I had floating around.

Bottom line is that we really need to implement some automated testing. I think that there are plenty of ideas and options around for the non-browser side of things; but it would be nice to double check things like the admin password (set by inithooks) allows login and stuff like that. The downside of that is that we will need to write a lot of tests which won't be that exciting... And we will probably still want to double check the screenshots (that the testing takes as it goes).

We probably really need to develop some clear goals, then build a plan of how we might attack this. Then we can zoom in closer on the details. Like I said, Anton has already done a fair bit of work on a PhantonJS based framework but it's been low on the priority list so keeps getting pushed back...

Regardless, thanks for your interest. Perhaps we need to start a new page on the wiki so we can develop the ideas and clarify the goals...?!

[a3s7p]

It's CasperJS, actually, which offers a superset of PhantomJS features (so more high-level, less boilerplate). Here it is.

To start with, it would probably be useful for me to lay out my vision of how this should work in practice somewhere, since I've written the bulk of code relating to this so far and faced quite a few practical issues along the way. A wiki page would do. Afterwards everyone can contribute.

[SiKing]

I work as an SDET. So testing getting a "lower priority" I am used to. :)
The project I was just on had their budget cut, so they decided to save money by dropping testing all together.

[Dude4Linux]

It might help to discuss the various use cases for a testing framework. I think @JedMeister is referring to a framework for testing appliances during or after the build process which would help him automate the process of creating a new release. Previously we've made use of the testssl.sh script to help improve appliance security. Recently, I suggested that Lynis was another tool that would be useful in ensuring that appliances were following security best-practices.
My first introduction to DevOps was at the 2013 DrupalCorn conference. Yes that's right, DrupalCorn. Here in Iowa, the three major Universities have all adapted Drupal as the CMS of choice. They take turns hosting a summer Drupal Camp. In 2013, I was investigating the idea of bootstrapping a small business which would serve Churches and other non-profits that had needs that were not being met in the marketplace. Having previously worked for a small business doing IT support, I felt that using DevOps and automation was the key to building a successful business. Two of the takeaways that I got from that 2013 conference were Ansible and Selenium. That was the impetus for starting work on the Ansible appliance.
For some months now, I've been using the Ansible appliance in conjunction with the LXC appliance to run simple tests on the 'proxmox' builds that @JedMeister provides. The great thing about Ansible is that it can easily run external scans such as testssl.sh or security auditing as in Lynis. I did find that Ansible's built-in html test capability was rather limited so that another appliance based on CasperJS, Selenium 2.0, or possibly both makes a lot of sense. I look forward to your proposal @qrntz.

Opps forgot to include the links:
testssl.sh https://testssl.sh/
Lynis https://cisofy.com/lynis/
DrupalCorn http://2016.drupalcorn.org/

[JedMeister]

@qrntz - Thanks for the clarification and awesome to have a quick look over your work to date. 😄 I agree that it may be good to share your vision/plan for this. Perhaps after you're done with the other priority stuff we're working on currently, you could briefly do that? Then even though we have higher priority stuff to be working on, perhaps there is room for others to keep working on this while we're busy doing other things? Just a thought...!?

@SiKing - Thanks again for your interest and offer of assistance. 👍

@Dude4Linux - Thanks for adding some clarification to the mix! You are correct, I am talking about some web UI testing stuff. The plan is for it to also take screenshots as it goes. This has a dual purpose. It will allow us to view any potential issues, as well as give us current screenshots so we can keep our appliance library screenshots on the website up to date and relevant. Actually it might be useful for you to share your work and the testing framework that you have already developed?! Perhaps we could/should integrate your work and Anton's into an additional appliance based on TKLDev?! I.e. add the LXC functionality, the Ansible functionality and the testing scripts etc? Perhaps that's overkill at this point, but it would be pretty damn cool to be able to build an appliance in the "proixmox" format, run it in LXC and then run a battery of tests on it! Also if you have already publicly shared your work and I either missed it or forgot about it, please accept my humble apologies. Finally as a total aside, for some reason I thought you were in the UK (Bristol springs to mind for some reason??) but Iowa is obviously in the US!?!

[Dude4Linux]

@JedMeister My vision for a DevOps suite of appliances is to keep blocks of functionality in separate appliances and provide API's that link them together kind of like Lego building blocks. The functional blocks can then be combined in various ways to suit the needs of the end users. A lot of cloud development, OpenStack, Docker, etc seems to be moving in this direction. I had to add some code to both the Ansible and LXC appliances to allow them to work together. In the future, I hope to do the same with the Jenkins and GitLab appliances. Using this model would mean that if someone prefers Salt, Puppet, or Chef over Ansible, they could develop an appliance that meets their needs and still take advantage of existing appliances. Someone who prefers Docker to LXC for container management could develop a Docker appliance with similar functionality. Currently TurnKey lacks the developer bandwidth to tackle all of the possible combinations, but I think you have done a great job in creating an open and welcoming environment for new developers.

Regarding TKLDev, I believe that the ability to build any of the supported formats including "proxmox" should be integrated into a future version of TKLDev. I think this is a natural extension of the "build" function. I agree that any built-in tests that run in the fab-chroot environment should be included there. Possibly a Lynis security audit belongs here as it is just a shell script that does not need to be installed. I took a look at Anton's @qrntz work and it looks to me like it could be adapted to capture screenshots during the build sequence. My feeling is that including LXC and Ansible in TKLDev would be overkill and harder for new developers to master. I do recommend that we look into including jinja2 templating into the next version of TKLDev and also adding back the ability to install patches during the build sequence. Several times I've had to resort to an overlay when I really wanted to just patch an installed file. Along that same vein, I'd like to see more of the pre and post script options from TKLPatch. Anton and I had a hell of a time getting some of the security changes applied at the right time for the 14.0 release. I know someone else is working on the next version of TKLDev, but I'd like to volunteer to help as well.

Regarding Ansible, I have shared some of my work on plays and playbooks in Gists and also included some in the 14.2 branch. https://github.com/Dude4Linux/ansible/tree/updates-for-v14.2/overlay/usr/share/ansible/containers I'm not particularly proud of any of them. Like most Ansible n00bs, I started putting everything into large playbooks that run like scripts. Now I'm faced with breaking them up and creating equivalent Ansible roles. My biggest stumbling block has been that the term 'role' carries meaning and some assumptions in my mind that I'm beginning to realize are not true in the way Ansible uses the term. I hope to get my head around it before it's time to release 14.2. As for the UK, you might have confused me with Jon Hawkesworth who's helping me add Windows support to the Ansible appliance. He's based in London (not sure about Bristol).

[JedMeister]

Let's make providing some level of automated testing to the v14.2 build part of the plan. Exactly what it will do and what level of testing it provides can be defined/clarified/decided at a later date.

Here's something that I think looks really cool: http://os-autoinst.github.io/openQA/

@qrntz - assigning you to this issue for now as I'd really like you to publicly state your vision (IMO without that this issue is somewhat blocked).

@Dude4Linux:

@JedMeister My vision for a DevOps suite of appliances is to keep blocks of functionality in separate appliances and provide API's that link them together kind of like Lego building blocks. The functional blocks can then be combined in various ways to suit the needs of the end users. A lot of cloud development, OpenStack, Docker, etc seems to be moving in this direction. I had to add some code to both the Ansible and LXC appliances to allow them to work together. In the future, I hope to do the same with the Jenkins and GitLab appliances. Using this model would mean that if someone prefers Salt, Puppet, or Chef over Ansible, they could develop an appliance that meets their needs and still take advantage of existing appliances. Someone who prefers Docker to LXC for container management could develop a Docker appliance with similar functionality.

Great vision/plan! I ❤️ it! 👍

Currently TurnKey lacks the developer bandwidth to tackle all of the possible combinations, but I think you have done a great job in creating an open and welcoming environment for new developers.

Thanks for the kind feedback. And you are definitely right about our current (shortfall of) "developer bandwidth" WRT to exploring all the possibilities. However if we have something that has been done and works (as per your vision) but with the flexibility of it to be used as sort of a "template" which could be used with alternate possibilities then that's be extremely cool.

Regarding TKLDev, I believe that the ability to build any of the supported formats including "proxmox" should be integrated into a future version of TKLDev. I think this is a natural extension of the "build" function. I agree that any built-in tests that run in the fab-chroot environment should be included there.

I agree. FWIW in case you weren't aware, it is already publicly available but isn't pre-integrated and is not particularly well documented. Have a look at buildtasks. One major shortcoming is that it (or more correctly; the OVA build) currently relies on the non-open source VMware OVFTool...

Possibly a Lynis security audit belongs here as it is just a shell script that does not need to be installed. I took a look at Anton's @qrntz work and it looks to me like it could be adapted to capture screenshots during the build sequence.

Hmm interesting idea. Definitely has merit.

My feeling is that including LXC and Ansible in TKLDev would be overkill and harder for new developers to master. I do recommend that we look into including jinja2 templating into the next version of TKLDev and also adding back the ability to install patches during the build sequence. Several times I've had to resort to an overlay when I really wanted to just patch an installed file.

So something like turnkeylinux/buildtasks#11 ?

Along that same vein, I'd like to see more of the pre and post script options from TKLPatch. Anton and I had a hell of a time getting some of the security changes applied at the right time for the 14.0 release. I know someone else is working on the next version of TKLDev, but I'd like to volunteer to help as well.

The more the merrier. Although I think it would be really good for us to try to communicate what/when we are going to work on different things. Perhaps we could even go as far as to develop some sort of read map for TKLDev? Otherwise if we're not careful we might end up treading on each others toes! 😄

Regarding Ansible, I have shared some of my work on plays and playbooks in Gists and also included some in the 14.2 branch. https://github.com/Dude4Linux/ansible/tree/updates-for-v14.2/overlay/usr/share/ansible/containers

Great. Sorry I missed that. TBH I haven't even played with Ansible so really need to spend some time on it. One thing I do really like though is that it's agent-less...

I'm not particularly proud of any of them. Like most Ansible n00bs, I started putting everything into large playbooks that run like scripts. Now I'm faced with breaking them up and creating equivalent Ansible roles. My biggest stumbling block has been that the term 'role' carries meaning and some assumptions in my mind that I'm beginning to realize are not true in the way Ansible uses the term. I hope to get my head around it before it's time to release 14.2.

Thanks for sharing your insight. I haven't barely even looked at Ansible yet so you've probably helped me avoid a pitfall! Thanks so much for sharing your vision and ideas. It get's me a bit excited really! 👍

As for the UK, you might have confused me with Jon Hawkesworth who's helping me add Windows support to the Ansible appliance. He's based in London (not sure about Bristol).

I obviously got you confused with someone; but I'm almost certain it wasn't him, it was from a long time ago. It's so weird I can still see "Bristol, UK" next to your avatar in my mind's eye... Obviously got my wires crossed somewhere. TBH I wasn't even sure where Bristol was until I googled it! Oh well... Might need to get some sleep!

[Dude4Linux]

@JedMeister I took a stab at implementing one of my suggestions above, namely being able to apply patches during the build. I thought I had been able to do that with TKLpatch but my memory was faulty. I do remember having the need to apply an upstream patch to fix a problem where the patch had not been backported. This was my first attempt to explore the TurnKey fab and modifying make files. I'll open a tracker issue and a pull request after I've had a chance to do some more testing. You can checkout the changes here https://github.com/Dude4Linux/fab/tree/add-fab-apply-patch