The purpose of this Industrial PhD project work is to design and develop a surveillance system capable of detecting anomalous incidents inside of a building. In this regard, the primary focus will be on office environments. Thus, given a floor plan of that office, it should be possible to represent contextual knowledge about this building prior to performing video analytics. This context will serve as the base to build an anomaly detection system on using visual data from a multi-modal setup of cameras. The latter being spread all across the office additionally allows for further knowledge enrichment by incorporating additional metadata into the process such as Access Control, for instance. To put it briefly: The surveillance system of a company should raise an immediate alarm in case an abnormal scene is detected. Those scenes may contain violence, the emergence of smoke/fire, access to facilities during unusual times of the day, and many more.