Calculator < Spreadsheet < R < ? for large n

It has previously been suggested that the introduction double entry accounting funded the renaissance. This seems quite illogical at first sight. How can a change to an accounting method produce more money? I suspect that the answer is that the new method reduced error and thus reduced the number of petty disputes. It also made theft harder for employees. With the new system, less checking was needed; trust was increased; supplier and consumer were in greater harmony. The point is: a new method, even for something as simple as adding numbers, can have a remarkable effect.

I recently watched my sister use a calculator on an Ipad to add up the numbers on a bill to check them. Initially, I was impressed by her fiscal diligence. Being me, I tried to convince her to use the spreadsheet that she has built in to her Ipad. She was quite unimpressed with this idea and kept using her Ipad calculator.

And then the numbers didn’t add up. Fortunately, I wasn’t so silly as to keep pressing her to use a spreadsheet and also fortunately, the calculator app being used had a history. It showed that a sneaky “zero times” had been inserted at the front.

The lesson is that, for small tasks, calculators (and mental arithmetic) are just fine but they don’t scale. As soon as the size of the problem increases (even slightly) one is soon left with repeating the calculation several times to check that you have the right answer.

It’s interesting to see that spreadsheet tools have a similar problem.

During a writing course that I teach to engineering and computer science PhD students, I ask them to take some real-life data, involving temperatures in different rooms of a house over time, and produce a meaningful graph. Since the data have more than two dimensions (room, time, temperature) producing a graph with most spreadsheet tools is difficult. Interestingly, very few of the otherwise well-educated students manage to produce a correct graph. Most often, they get a graph like this:

This particular graph is wrong because it doesn’t distinguish between different rooms and truncates the date value incorrectly. Spreadsheet tools produce a wrong graph easily. Unfortunately, it is almost impossible to get such tools to produce a correct graph.

R, and in particular the ggplot2 library for producing graphs, are much better tools for this task. Here is one solution as R code.

x <- read_csv("getData.csv", skip = 2,
              col_names=c("nodeId", "house","room","time","value"))
y <- mutate(x, time= as.POSIXct(time, format="%d/%m/%Y %H:%M:%S", tz="GMT"))
ggplot(y, aes(x=time, y=value, colour=room)) +
  geom_point() + geom_line()

This code looks complicated but it’s been created step by step in Rstudio, a tool that makes it easy to construct the code and test each piece as you go.

Here are the key steps:

  1. read the data into memory
  2. convert / massage data
  3. plot the massaged data

Step one is to read in the data into memory. I prefer read_csv from the readr package because it’s fast and it doesn’t make too many assumptions or try to interpret the data. A first attempt might look like:

x <- read_csv("getData.csv")

This almost works but the extra header lines cause a problem. skip = 2 gets rid of these but loses the header information so a simple fix is to manually specify the column names with col_names.

The next step is to convert the time from a string to a time / date format. R has a number of ways of storing times but POSIXct, which stores date and time as seconds since 1st Jan 1970, is usually a good choice. Note that when converting, you need to say what timezone is involved. Since the data was gathered in the UK, GMT is probably a safe bet.

The conversion requires a format string. This is another improvement over the spreadsheet tools, which guess the format and sometimes wrongly choose M/D/Y when the date is really D/M/Y.

With R, we have a variety of ways to update a column. The following might look simpler to some:

x$time <- as.POSIXct(x$time, format="%d/%m/%Y %H:%M:%S", tz="GMT")

This uses $ to reference a data frame column by name.

This syntax is a bit unreadable though and dplyr offers a nice alternative. For this short script, either will do.

The final step is to plot the data. R has in-built plotting tools but ggplot2 provides much more flexibility. We start by telling it which data frame and what axes. Note that the axes parameters go inside aes() (short for aesthetic).

ggplot(y, aes(x=time, y=value, colour=room)) + geom_point() + geom_line()

ggplot() on its own doesn’t produce a plot. For that, we need to add a layer. Here, we’ve added two: one that produces a point for each data point and one that draws lines between the points. The result looks like this:

Temperature plot created with R and ggplot2


Note that we’ve still got work to do here: for example, the y axis might be better labelled “Temperature (deg. C)”. However, it is a good start and much nicer looking that anything that I’ve been able to produce with a spreadsheet tool alone.

Wrapping up

My conclusion is not that one needs to stop using calculators or spreadsheets but rather that one needs to be aware that as n grows larger (where n is number of items, number of dimensions, or complexity of the task), basic, generic tools fail and more sophisticated and specialised tools are needed. When the problem gets hard, don’t forget to upgrade the tool.

Daily planning

I am terribly easily distracted. I can start my day with a firm plan to get on with writing a scholarly paper and end up playing all day with some programming tool that I have just discovered.

Why is this a problem? Well, sometimes it doesn’t matter that I occasionally get distracted and I am able to string together sufficient periods of focus and concentration that I perform the work that I need to. However, it sets a bound on what I can do in a certain period of time that is lower than my real ability.

The solution? Make a plan each day. At least, that’s what the self-help books will say. And they’re not wrong but the quality of the plan and the way that you react to your plan can make a big difference. Let me start by telling you about my planning approach – which is roughly sketched in the mind-map here.

The A5 notebook

Let’s go through this one bit at a time. Start by getting an A5 spiral bound notebook. The A5 size is good because it doesn’t clutter your desk and is big enough to hold your daily plan. Spiral bound is better than other sorts of bindings because it means that you keep a record (better than tear-off sheets) and that it opens flat without having to hold it open.

Divide the page vertically in half. You can do this by folding the page to get a top-to-bottom crease. Another way is to rule a line. On the lefthand side of the line we’ll put the day. I usually start at 8:30 but you can start as early or late as you like. With an A5 page, I reserve one line per half-hour slot by skipping the top line and then writing 9-skip-10-skip … 5, from top to bottom.

Book fixed slots

There are some things in your day that will happen at a particular time and other things that might move about. For example, the time that you arrive in the morning is generally fixed. Similarly, it is good to fix your lunchtime and leaving time. Appointments that have been booked are generally also immobile. These fixed slots can be written into your daily schedule. I generally use a brace sign “}” to indicate slots that occupy more than a half-hour.

So, start by writing in your meetings, lunchtime, start and end times. Add a fifteen minute slot for the planning session (generally at the start but can also be done at the end of the day). Add a further fifteen minute slot for review at the end of the day.

This forms the skeleton of your day.

Schedule your e-mail processing

If you are like me, you are easily distracted by your e-mails. Try scheduling slots when you can look at e-mails. Avoid scheduling e-mail slots at the start of the day, where they can influence your planning process. Instead, try scheduling them, say before lunch and just before going home. This has the effect that there is a definite limit to the amount of time processing e-mails.

E-mails can hijack your priorities. Don’t let them! Keep your e-mail program shutdown until the allocated slot.

If this seems like an advanced concept that maybe you’re not ready for, that’s fine. But when e-mail processing starts to rule your day, it’s time to tame the tiger and restrict how much time you spend on it.

The Task List

The list of tasks goes on the right hand side of the page. You can start writing close to the crease or ruled line; the annotations that we will add later can go on the left of the line.

Step one – copy forward

Step one is to copy forward any tasks from the previous day. This shouldn’t be done dully. If a task is being constantly copied forward without being completed day after day, it indicates a problem. Perhaps the task is not really that important? Perhaps it is waiting on something else? Perhaps it is simply too hard?

There may not be simple solutions to such sticky tasks, however here are some things to try:

  1. Do it. Sometimes a much-put-off task is only seemingly scary. Once you start to do it, you find that you get it done quickly and without fuss.
  2. Dump it. Perhaps it is not as important as you thought it was.
  3. Delegate it. You may not be the right person for a certain task. Be ready to seek help from someone else.
  4. Schedule it. Is it waiting on something else? Will it soon become easy to do? Is it being pushed back because it is a low priority task when you have a lot of high priority work to do? Sometimes the answer is to schedule a reminder for next week, when perhaps there will be more time.
  5. Break it down. It may be too large a task to tackle in one day. You may be making progress but because you are not ticking off to do items for this task each day, it feels like no progress is being made. You can avoid this by treating it like a project with sub-tasks. Sometimes even the sub-tasks will need to be broken down into sub-sub tasks before it feels like you are making progress.

Step two – copy from master list

Step two is to copy items from your master task list. Personally, I keep a long-term or master list of tasks on my computer but it can also be in paper form, if you prefer. Again, you must choose these tasks carefully so that you don’t avoid difficult tasks but also don’t overload your task list on any one day. We’ll need to check for overloading later on.

Step three – break down large tasks

Step three is to break down larger tasks into smaller tasks. On a day planner, the task “write a paper” is too large a task to do in one day. It needs to be broken down into smaller chunks.

Breaking a large task into smaller chunks can help in a number of ways: It helps give a sense of progress; it makes it easier to estimate how long the task will take overall; it makes it clearer how to go about a task.

Step four – assign priority

Priority helps determine the order in which things should be done. That’s important because, without making a conscious decision about priority, all the easy tasks will get done first and the hard tasks will be left for tomorrow.

The first step is to decide which tasks need to be done today without fail. These are “A” tasks. If you never have A tasks, don’t worry. It’s not a sign that what you do is not important. All other tasks are “B” tasks. There’s really no need to have further categories but as with any of the suggestions here, you can do as you please. (And please write a comment if you have a useful improvement!)

Within the A and B tasks, we now need to set a priority ordering.

Priority has to come from your overall priorities. Do you know what they are? They might be 1. Write X journal papers per year; 2. Help out other team members; 3. Teach class on Y. etc. Whatever they are, it’s useful to have your priority list handy when you are prioritising.

When you’ve finished this stage, all tasks will have a letter (e.g., B) and a number (e.g., 3) to make up a code (B3) that we write to the left of the task.

Step five – estimate task duration

Identifying how long a task will take can be hard. It’s a common mistake to underestimate how long something will take us.  There are two solutions to this problem. First, break-down large tasks into smaller chunks. Usually these smaller chunks will be easier to estimate. For example, I don’t know how long it will take me to write a paper but I can have a pretty good guess at how long it will take me to write an abstract. I can guess even more accurately if the task is “rough first draft of abstract”.

Second, get some practise and reflect on the results. As with many things, practise will improve your ability. For example, the first time you estimate how long for a rough draft of an abstract, I might estimate 30 minutes (or 1 pomodoro – see below). However, when I came to write that abstract, I realised that I need to spend 30 minutes brainstorming and organising ideas before I was ready to put finger to keyboard. So the real time, was more like 2 pomodoros.

So what are these pomodoros? The term comes from the Pomodoro Technique, which is a wonderful method for staying focused and organising your time. The day planning technique used here is intended to be used in conjunction with the Pomodoro Technique. In PT, a “pomodoro” simply refers to 25 minutes of work followed by 5 minutes of rest.

When you are finished with this stage, you’ll have a duration for each task. Durations less than 1 may mean that you are breaking down your tasks into too small chunks. Durations larger than 6 may indicate that the task is too large.

Carrying out the plan

Once you have a plan, you can start carrying it out. As you do so, note down in your timeline for the day which task was done when.  This will help later when you come to review.

Keep to your priority ordering, doing the A tasks first.

Try following the pomodoro technique of taking five minute breaks every 30 minutes.

Finally, be aware of your limits. If you have a day full of meetings, don’t expect yourself to also achieve on the task front.


Before retiring at the end of the day, review progress on tasks. Were the meetings fruitful? Did you progress with the tasks? Even if not everything went to plan, don’t fret. Find three things to be happy about.

Did you do something nice for someone? Note it down. Also you can record your progress on forming good habits, such as regular meditation or exercise.

If you like to have an overall assessment, draw a smiley (or frown-y, or wobbly mouthed) face to reflect on how you feel.

At the end of the week, review progress over the whole period. Again, try to focus on the positives.

Finally, feel free to adapt the approach to suit you. And when you come up with a really useful change, don’t forget to tell me about it!


My planning approach is largely based on a paper-based approach originally advocated by Priority Management. My approach differs in a number of respects but is derivative.


Why don’t you pre-print sheets with this layout?

This is a personal preference but there are several reasons:

  1. Separate sheets would need to be bound into a book to form a permanent record
  2. Printed or photocopied paper doesn’t absorb ink well and has a rough feel. Plain paper is just so much nicer to write on.
  3. Photocopied sheets tend to be A4, which is unnecessarily large and clutters the desk.

What are you looking up here for?

When I was young, a favourite activity on the way home from school was to stop into a magic shop in a tiny arcade in the centre of Melbourne. Sometimes, if we were lucky, the owner would show a group of us some trick to encourage us to spend some money. I remember specifically standing in that shop one afternoon, agog with its delights when I finally spied a sign on the ceiling that said “what are you looking up here for?”

So I can start posting on this blog only if I am completely honest with the reader and admit that I cannot pretend to offer anything yet. Please look in on me some other time and there might be something here of value … but not just yet.