Technology, nowadays, is ruling the world. Computers are made artificially intelligent. Researchers are trying to give all humanly capabilities to computers in an effort to improve people’s lives. Till now, the computers can sense using sensors and get involved in interesting conversations but lack the capability of understanding a real scene like humans. Inspired by the significance of human-computer interaction, we have presented a technique to recognize human-human interactions to allow machines understand the happenings in the outer environment and act accordingly. Our proposed recognition system aims at recognizing six basic human interactions i.e. Hand shaking, hugging, kicking, pointing, punching and pushing. Responses may be taught after correct prediction of these interactions. The proposed system should be used in security systems, robbery, uncertain violence detection, responsive toys, virtual games and even sports.