To examine the features of complex visual world, sensor technology merged with objects characteristics to scenes well. These scenes understanding are highly demanding task in different domains of visionary technologies like autonomous driving, generic object detection, sports scene identification and security. In this project, we proposed a novel statistical segmented framework that can learn robust scene model and separate each object component. Then, each component is used to extract geometrical features that concatenate extreme points features, orientation and polygon displacement values. These features help in object detection and Gaussian Naïve Bayes is used for the scene recognition. The experimental evaluation demonstrated the proposed approach over UIUC Sports and 15 Scene datasets that achieved scene recognition rate of 85.09% and 82.65%. The proposed system should be applicable to different emerging technologies such as augmented reality scene integration, GPS location finder and visual surveillance which recognized different locations/objects to understand real world scenes.