Tilca Technology Technology updates and thoughts from Tilca

Bitesize Improvement 2: Tester

by Art

Posted on Wednesday Apr 20, 2016 at 08:08PM in Technology

Following the previous entry in this series, here are some suggestions for improving your testing skills.

Domain understanding

  1. Can you explain the point of the application you're testing, in about one minute? What are its benefits for the users? What about for the people who are paying for the development and testing? (These two groups are not always the same.)

  2. What would happen if the application didn't exist? Could its work be done by hand? If so, try doing the work by hand. You'll be forced to think through each step in the process which should help you think of test cases.


Your remit might not extend to usability (many projects delegate this to a different role, such as the UX designer), but using an application in different ways that reflect real life use can expose flaws.

  1. Watch some actual users of the application, if it's already in production. You don't have to quiz them or do it in a structured way.

  2. Watch some actual users of a different application. People use computers (and tablets, and mobile phones) in different and personal ways. You might never have watched someone else use their mobile phone for any length of time or to complete a specific task. Other users do strange things like rotating their tablet in the middle of entering text. Maybe you need to test that.

Test data

Predictable, easily regenerated test data will let you get on with exploring the system instead of spending time setting up the system under test, which can be laborious and error-prone.

A little help from the developers of the application will go a long way here. I recommend they create a special API that allows some canned data to be recreated. This API could also have a way to reset the system into a known state, deleting existing data and clearing any caches.

Rules of Refactoring Club

by Art

Posted on Monday Aug 03, 2015 at 09:09PM in Technology

with apologies to Fight Club

  1. You do not talk about refactoring club

    Obviously, as a developer you will need to talk to other developers and architects about refactoring. But talking to non-technical people from the business (e.g. product owners) about refactoring is unlikely to be productive. Unless they used to be developers themselves, they won't appreciate its importance.

    So how do you persuade them to allow you or your team to get on with it?

    My suggestion: don't persuade, or even discuss, with the business. Just get on with it. Hopefully, you're in an agile team, committing to certain features per sprint.

    • Include a certain amount of slack in your estimates.
    • If there is a lot of refactoring to do, then include a large amount of slack. (Hint: how did you end up here?)
    • Try not to include refactoring items as 'stories'.
    • If there is so much refactoring that you need to track and prioritize what needs doing, use a different tool from your feature tracker. Do not waste your business people's time by showing it to them.

    The earlier you do this, the better. Get used to including slack. There is always pressure to deliver features, but in reality, business people have very little idea what rate of progress to expect from a development team. At the start of an engagement, you have a great opportunity to manage their expectations.

    You might be a little uncomfortable with this approach; it could seem dishonest. But you don't discuss the intricacies of git rebase vs. merge with your business people; your car mechanic does not ask your permission to unscrew each bolt. In my experience, it's the best way to ensure the business actually gets what they want from their software. Keeping quality high by constantly refactoring will allow you to deliver features more quickly as the project grows.

    Martin Fowler takes a similar line in the discussion towards the end of this video. (In fact, just go watch the whole thing; this can wait)

  2. You do not talk about refactoring club

    Don't misuse the word refactoring. Martin Fowler can have the first and last word on what it means:

    Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.

    It does not mean 'change' or 'fix'. If you need to change or fix code, just admit it (unlike the previous point, I think it is fine to discuss this with people from the business.)

    I've heard people say 'I refactored the screen to show X' or 'We need to refactor this API to allow Y'. What you're doing there is changing those things, because they're externally different afterwards!

    Refactoring has a very domain-specific meaning, and it's hard to create satisfactory analogies (please leave yours in the comments, if you have one). Maybe that's why it's so hard to explain how important it is to non-programmers.

    The meaning is lost if you use the word incorrectly, and the implication of past failure is hard to avoid if it starts to mean 'fix'.

  3. One refactoring at a time

    Within reason, obviously. If you're working in completely separate areas, it's not a problem.

    Two people refactoring the same area at the same time will quickly figure out it's a bad idea when their source control starts spewing conflicts. If your team's communication is good, this won't arise, but large projects always have problems. Don't let well-intentioned refactoring get a bad name just because your developers aren't communicating well.

    More subtly, make sure you finish your refactorings. There's nothing worse than some originally bad code, copy-pasted around, only some of which gets tidied up. Now the unfortunately soul trying to understand it will need to figure out two lots of code.

    Unfinished refactorings sat on a branch are worse, because they will begin to rot as changes are introduced to main-line code that's affected; the refactoring might need to be applied to fresh code.

  4. Refactoring will go on as long as it has to

    There is always more refactoring that could be done; only you and your team can judge when it's necessary.

    Be especially careful of refactorings deemed necessary for performance and scalability (this blog post is mostly concerned with refactoring for maintainability). I can't explain why, but in my experience, these kind of proposed changes seem to bring out developers' egos and encourage unproductive, non-consensual discussion. Having a rudimentary performance test framework in place early on in the project will help here.

  5. If it's your first night at refactoring club, you have to refactor

    In agile software, design emerges throughout the development process rather than being handed down from on high. It doesn't emerge spontaneously: refactoring is what makes it happen.

    Encourage everyone, even junior developers, to refactor constantly. They'll be able to develop their design skills, in a relatively safe way (you do have lots of automated tests, right?). It's a great way to spot who has potential to be a 'star' programmer.

    Code reviews are another useful tool for pointing out opportunities for refactoring, as long as you don't overdo it. Sometimes it's best left for another merge request.

The YAGNIsh Index

by Art

Posted on Saturday Jun 06, 2015 at 02:40PM in Technology

When you're in the early stages of designing software, "You Aren't Gonna Need It" ought to be ringing in your ears. After all, you don't want to end up like this (courtesy of Jeff Atwood): What the hell have you built

If your software development is truly agile, your architecture can evolve quickly in response to changing technical requirements. So start simple. But what do you start with? Some tech choices are so ubiquitous that you can be pretty sure you are going to need them. I give these a low "YAGNIsh" score.

In my experience, other technologies will tempt you, but should often be deferred until later in the project, due to a variety of reasons (mostly the carrying cost of the extra complexity). These are highly "YAGNIsh". Of course, this is only a guideline: your experience might tell you that you'll definitely need, say, a rules engine. But if you've never used one before, beware.

Workflow engine

This is a great example of something you can put in later, if you really need it. Almost all software has some form of workflow, even if it's just "reset password", so it's tempting to think that a workflow engine will do a better job than bespoke code. But you and your team are going to spend weeks understanding and then integrating that engine into the rest of your code. In my experience, the engine won't do exactly what you need either, so you're going to have poke around in the source (if you're lucky enough to use an open source one).

YAGNIsh score: 10/10

Source control

Here's an obvious counter-example. In fact, everything from Joel's 12 Steps has very low YAGNIsh. If you have experienced developers, source control won't be an overhead. Use source control right from the start!

YAGNIsh score: 0/10

(MongoDB|Cassandra|other NoSQL flavour of the month)

Are you sure a bog-standard, boring RDBMS won't suffice? Relational databases can process thousands of transactions per second. Postgres can store 32TB in a single table. Even the mighty Stack Overflow runs on SQL Server (with some caching, obviously)

"But, performance!". Yes, it might turn out to be a problem, so you should be measuring performance as early as practicable. It will help to gather back-of-the-envelope numbers on likely usage patterns. (You'd be amazed at how many projects do not do this.)

Of course, sticking with an SQL store has other advantages in terms of data integrity (such as a proper schema) and ease of reporting.

YAGNIsh score: 9/10

Message queue

If you have a distributed system, a message queue could well be a good idea. Maybe you have a third party component that's frequently unavailable or can only cope with a certain load, and you want to have automated retries.

Just remember that asynchronous programming is harder and less intuitive than the old-fashioned synchronous way. You have to deal with correlation and timeouts. Your stack traces are split in two or three. How long does it take a developer to reset the state of their message queue?

If you're not sure, start simple. Maybe instead of a queue, you could just use a thread pool to prevent blocking.

YAGNIsh score: 5/10

Database schema management

Free tools like Liquibase and Flyway manage your database using version-controlled scripts. Scripts can be in XML or plain DDL depending on your preference. You can use these tools on a fresh database, or apply them to an existing one that's already in production.

Like source code version control, these are essential, once you are in production. Before your first release, you might find it simpler to recreate the database each time with a single script (version-controlled along with the rest of the source code), since the data in the database doesn't matter. But it's a good idea to get your developers some experience with the tool by having a few practice releases.

YAGNish score (pre-release) 6/10

YAGNish score (production) 0/10

Third party identity management

If "social" is a key part of your application, integration with OpenID may actually save you development effort since you do not need to deal with saving usernames and passwords. You won't need to worry about passwords being stolen, or users resetting passwords they've forgotten (which is surprisingly complicated to get right).

On the enterprise side of things, you may not have much choice if your client wants single-sign-on across an existing corporate estate, so you might as well take the pain early on.

YAGNIsh score: 3/10

Let me know your favourite YAGNIsh technology in the comments!

Bitesize Improvement 1: Developer

by Art

Posted on Saturday May 16, 2015 at 11:00PM in Technology

The hallmark of successful people and organisations is continual improvement. Here are some small suggestions for improving yourself as a developer.

Learning a little bit on a regular basis is a great way to build your confidence and avoid stagnating. So don't try to learn everything at once; just pick one thing that you think looks interesting that you don't already know.

I've put these in order of complexity, with 1 being very easy and intended to be an introduction to the subject.

Regular expressions

Yes, we've all heard the joke.  But a simple regular expression can save you much trial-and-error. If you've never used one, you're missing out.

  1. Capturing group

    This is a very simple application of regexes. For example,

    a (blue|red) car
    matches both "a blue car" and "a red car"
  2. Find/Replace

    The syntax for find and replace varies between regex flavours. Java uses $1 for the first match, $2 for the second, etc.

    So you could pimp up the car in the previous example with a replacement string

    a flashing $1 car

    which would give 'a flashing red car' or 'a flashing blue car'

  3. Negative Lookbehind

    This is quite advanced, and useful when you are trying to avoid matching the wrong thing. The excellent regular-expressions.info site explains it better than I can.

Refactoring Tools

Learn your IDE's refactoring tools.

  1. Rename

    Renaming classes, methods, and variables is a simple but powerful way of keeping your code base sane. It requires very little effort, once you've learned your IDE's keyboard shortcut. If you don't know it yet, here's a couple of links to help you out.

  2. Extract Method

    The 'Extract method' refactoring is your trusty sword in the battle against duplicated code. A statically typed language such as Java or C# helps here, but modern IDEs such as PHPStorm and PyCharm (both from the magnificent JetBrains) do a pretty good job with dynamic languages.

    The opposite refactoring, 'Inline method', is less frequently used, but still useful. For example, during a complicated refactorings you might be left with a short method that is only called in one place. Feel free to inline it, if that makes the code more readable. You can always extract it again later.

  3. Encapsulate field

    OO purists will insist that member variables of a class cannot be accessed outside of that class, and that you ought to write tedious get and set methods. Well, your IDE probably does this for you with its 'Encapsulate field' refactoring.

    In my opinion, the availability of this refactoring makes accessor methods less important, since you can always introduce them when you need to. (This is only true when you control all the code, and not, for example, in libraries and APIs consumed by others. But in that case, it can be argued that you ought to be providing interfaces, not concrete classes.)

Microservices: 3 questions for your project

by Art

Posted on Friday Mar 06, 2015 at 12:00AM in Technology

Microservices has taken over the world in the last couple of years. The concept of loosely coupled, highly cohesive components is an attractive one, but it can have its disadvantages. If you're going down this route, then thinking about the following questions will make sure your project avoids the pitfalls.

What scale are you aiming for?

Software can be challenging to scale along two axes: volume of use, and complexity of behaviour. Small services can help with both, but only if done right. If you have low volume of use, and your application is simple, it may be hard to justify the overhead of microservices.

When performance testing identifies a particular service that is a bottleneck, it should be easy to scale it up, without needing to change the rest of the system. (This is the key advantage of microservices.) For this approach to work, you need to be able to get sufficiently fine-grained data from your performance testing tools. Get familiar with one early in the project; you should not leave this until the system is live.

Behavioural complexity is more difficult because you do not want to avoid a big ball of mud, only to find yourself with a distributed big ball of mud. Arguably, concentrating on the logical separation of microservices (rather than physical) will help to drive good design. For example, you may choose to have logically separate microservices that just happen to be deployed in the same container at runtime, using normal method calls between them rather than the more conventional REST/HTTP calls. This will probably make it easier to reason about the system at runtime.

Who owns the microservices

Software is written to solve the needs of whoever is paying for it ('the business'). In large organisations, different lines of reporting are frequently responsible for different applications. When these applications need to talk to each other, organisational politics can intervene. This was one of the motivations behind Service Oriented Architecture, before that term was co-opted by companies selling Enterprise Service Bus software.

If in your project, the microservices are all owned by the same business function, you are going to have an easier time making changes to them, since you will not have to negotiate with other parts of the organisation.

Conversely, the fun really begins when your microservice is a client or server of some other business function's services. You will need to spend more time thinking about versioning and support, and more time communicating. Good documentation, which is always important, becomes vital. Months and years down the line, you need to be able to look at a given service's logs and determine whether it is actually still being used; it won't be necessarily be sufficient to ask the team responsible for it.

How are you dividing your services - functional or technical?

Functional services that handle a specific part of the business domain (e.g. Address) are a natural fit for microservices because they can be easily understood even by non-technical people involved in the project.

It is also possible to split services in a more vertical fashion. For example, a microservice for database access, a microservice for authorization, etc. A user's request would pass through multiple such services while being processed. This is more in line with the traditional layered approach to application development.

There is a potential problem with adopting both on the same project, because that will result in an explosion in the number of services: N functional areas * M layers.

The functional approach fits more into an agile workflow where a single team is responsible for the delivery of a feature. Technical services, if they are stateless, can be refactored into common modules that are incorporated into each microservice.