As a particular case of the Finite Mixture Model (FMM), Rayleigh Mixture Model (RMM) is considered as a useful tool for medical ultrasound (US) image segmentation. However, conventional RMM relies on intensity distribution only and does not take any spatial information into account that leads to misclassification on boundaries and inhomogeneous regions. In this paper, we propose an improved RMM with Neighbour information (RMMN) to solve this problem by introducing neighbourhood information through a mean template. The incorporation of the spatial information made RMMN more robust to noise on the boundaries. The size of the window which incorporates neighbour information was resized adaptively according to the local gradient distribution. We evaluated our model on experiments on synthetic data and real US images used by High-Intensity Focused Ultrasound (HIFU) therapy. On this data, we demonstrated that that the proposed model outperforms several state-of-art methods in terms of both segmentation accuracy and computation time.