The introduction of the smart home has been seen as a way of allowing elderly people to lead an independent life for longer, making sure they remain safe and in touch with their social and care communities. The assistance could be in the form of helping with everyday tasks, e.g. notifying them when the milk in the fridge will be finished or institute safeguards to mitigate risks. In order to achieve this effectively we must know what the elderly person is doing at any given time. This paper describes a tiered approach to deal with recognition of activities that addresses the problem of missing sensor events that can occur while a task is being carried out.