In my role as a consultant, I am also a part-time team lead. I handle this by having one-to-one meetings with my team members as well as group meetings. In one of my one-to-one meetings, my colleague asked me when it was prudent to start automating tests.
I gave my usual answer; It depends.
We went over it for a while. The problem was that it was a front-end project, and it was still in the early stages. We discussed the issue and what you could do to avoid getting affected by the changes. Who should update the tests when they fail? How often do you want them to run? Do you have a CI/CD pipeline? How important is the test? Et cetera.
As usual, I did not provide a straight answer. I hardly ever do. My goal is to teach you to think, not provide you with the solution. I know what I would have done, given the information I had. But that knowledge comes from many years of trial and error. I might even be wrong. “Teach a man to fish”, and all that.
Give a Man a Fish, and You Feed Him for a Day. Teach a Man To Fish, and You Feed Him for a Lifetime.
- unknown
So what is the answer to the question?
Well, it depends.
Should you automate this particular test?
Not all tests are worth the trouble of automating. It might feel good doing it, but is it necessary? Is it important? How often will I need to update it? Does the feedback the test provide give value?
How often do the tests run?
This question is interesting because you are investing resources in writing the test. Let us assume the test takes 1 minute to run manually, and if it runs once per sprint, the return of investment (ROI) is longer. Let us assume, again, that you spend 60 minutes writing the test. Then it will take 60 sprints before you get your investment back.
On the other hand, if you have a CI/CD-pipe and all code written and checked into the repository is built, tested and released to the end customer, you would only need the project to build 60 times for it to start paying off. Then you reach your ROI breakpoint much faster. Your ROI is shorter.
How stable will the test be?
Automated front-end tests with frameworks such as Selenium WebDriver and Cypress are notoriously unstable. They are expensive to maintain, break down often, and tests are flakey. We can invest in fixing that. But as stated before, the cost can be high.
One thing you can do to increase the stability of the tests is to add unique identifiers to the elements with which you are interacting. Either you do this yourself or ask a colleague for help. Often it is easier for a developer to do it since they know where in the code you need to add the identifier. I prefer using a custom attribute on the elements since id and classname may be subject to change. But if you have a data-test attribute, everyone and their mother knows it is for a specific purpose and will not tamper with it.
If you write tests on a lower test level (Service or Unit), many of the problems inherent with GUI testing will magically disappear, and your environment is more stable.
Who will maintain the tests?
Will you be the only one maintaining the tests? What will happen when you are on leave or have moved on to another team or assignment?
I usually try to put the responsibility of the quality of the product on the team. Investigating and solving issues with failing tests is part of that. I often start with, “you break it, you fix it”. Of course, I will be there as a helping hand to fix it if needed. However, I never do the job myself; we pair-program.
The “you break it, you fix it” mentality is extra effective if you have a CI/CD-chain. Then the one breaking the test is alerted and can mend the problem shortly after they have worked on the ticket that broke it. It also blocks releases of the product, and that usually is a bad thing. We want our code to be released as fast as possible to get our investment back.
How prioritised is the particular flow?
Have you looked at other flows for the application? Is the one you are looking at the most important one? Are there other tests that are more suited for automation and give more value?
In conclusion
In my opinion, it is never too early to start automating. The question here is, considering everything we have talked about above, is it worth it?
What did I miss?
Happy hunting!
