Closing the application from the X in the top right-hand corner does not fully close Xbench, it only minimises it.

Shut the application down either by going to Project > Shutdown Xbench or by right-clicking the Xbench icon in your system tray and selecting Shutdown Xbench.