Smart phones, as one of the most important platforms for personal communications and mobile computing, have evolved with various embedded devices, such as cameras, Wi-Fi transceivers, Bluetooth transceivers and sensors. Specifically, the photos taken by a smart phone has the approximate or even equivalent image quality to that of a professional camera. As a result, smart phones have become the first choice for people to take photos to record their ordinary life. However, how to manage thousands of photos on a smart phone becomes a challenge. In this paper, we propose a new architecture in terms of Smart Photo-Tagging Framework (SPTF) to manage the substantial number of photos taken by smart phones. In particular, our SPTF collects the ambient data obtained from various embedded sensors on a smart phone when a photo is taken. After processing and analyzing the ambient data, SPTF can accurately record both the ambient tags and the face tags of the photo, which will be used for auto-tagging photos and searching photos. We also implement SPTF and verify its effectiveness by conducting a number of realistic experiments.