WO2005124594A1 - Automatic, real-time, superimposed labeling of points and objects of interest within a view - Google Patents

Automatic, real-time, superimposed labeling of points and objects of interest within a view Download PDF

Info

Publication number
WO2005124594A1
WO2005124594A1 PCT/IB2005/051924 IB2005051924W WO2005124594A1 WO 2005124594 A1 WO2005124594 A1 WO 2005124594A1 IB 2005051924 W IB2005051924 W IB 2005051924W WO 2005124594 A1 WO2005124594 A1 WO 2005124594A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
label
interest
superimposing
location
Prior art date
Application number
PCT/IB2005/051924
Other languages
French (fr)
Inventor
Elmo Marcus Atilla Diederiks
Bartel Marinus Van Der Sluis
Johannes Henricus Maria Korst
Hubertus Maria Rene Cortenraad
Original Assignee
Koninklijke Philips Electronics, N.V.
U.S. Philips Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics, N.V., U.S. Philips Corporation filed Critical Koninklijke Philips Electronics, N.V.
Publication of WO2005124594A1 publication Critical patent/WO2005124594A1/en

Links

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3626Details of the output of route guidance instructions
    • G01C21/3647Guidance involving output of stored or live camera images or video streams
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/006Mixed reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/20Scenes; Scene-specific elements in augmented reality scenes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10Character recognition
    • G06V30/14Image acquisition
    • G06V30/142Image acquisition using hand-held instruments; Constructional details of the instruments
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B29/00Maps; Plans; Charts; Diagrams, e.g. route diagram
    • G09B29/003Maps
    • G09B29/006Representation of non-cartographic information on maps, e.g. population distribution, wind direction, radiation levels, air and sea routes
    • G09B29/007Representation of non-cartographic information on maps, e.g. population distribution, wind direction, radiation levels, air and sea routes using computer methods
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B29/00Maps; Plans; Charts; Diagrams, e.g. route diagram
    • G09B29/10Map spot or coordinate position indicators; Map reading aids
    • G09B29/106Map spot or coordinate position indicators; Map reading aids using electronic means

Definitions

  • the present invention relates to superimposing, onto an image, labels of points and objects appearing in the image. More particularly, the present invention relate s to automatically in real-time determining if a predetermined point or object of interest appears in an image and, if so, superimposing the corresponding label onto the image. It is known to determine location and orientation of a camera taking pictures. The registration of location for the purpose of easy browsing and retrieval of the pictures at a later time is described in U.S. Patent Nos. 5,506,644 to Suzuki et al. and 6,222,985 to Miyake
  • the present invention has been made to address the above -noted shortcomings in the prior art. It is an object of the invention to provide a device, a method operabl e on the device, and a computer program for performing the method, wherein a device usable with an image - viewing apparatus marks, automatically in real -time, an image captured from a viewpoint. The image is bounded in two dimensions by a field of view at a particular orientation. The device communicates with a storage medium for storing location information of a point of interest and/or an object of interest.
  • the device includes a processing unit configured to, automatically in real-time, update a value of at least one parameter that defines the capturing of the image from the viewpoint.
  • the processor also determines, based on the location information and at least one updated value, whether the respective point or object is disposed within the field of view at said viewpoint at said particular orientation.
  • a label superimposing unit configured to, automatically in real -time, based on the determination that the respective point or object is so disposed, and under control of the processing unit, superimpose a label onto the image to identify the respective point or object. Details of the invention disclosed herein shall be described with the aid of the figures listed below, wherein: FIG. 1 is a schematic diagram of a real-time label superimposition system in accordance with the present invention; FIG.
  • FIG. 2 is a how an image with superimposed labels appears to the user through the viewfinder of a camera in accordance with the present invention
  • FIG. 3 is a conceptual diagram of a two-dimensional field of view in accordance with the present invention
  • FIG. 4 is a flowchart of a process for superimposing a label for a point/object of interest within the viewed image in accordance with the present invention
  • FIG. 5 is depiction of a sub-process of mapping a stored shape to the viewed image in accordance with the present invention
  • FIG. 6 is a flowchart of a sub -process for refining label placement in accordance with the present invention using the sub-process depicted in FIG. 5;
  • FIG. 5 is a flowchart of a sub -process for refining label placement in accordance with the present invention using the sub-process depicted in FIG. 5;
  • FIG. 5 is a flowchart of a sub -process for refining label placement in accordance with the present invention using the sub
  • FIG. 7 is a flowchart of a method for retrieving labels and associated metadata to be displayed upon playback or re -presentation of recorded images; and FIG. 8 is a flowchart of a process for retaining label information for subsequent representation of the stored images.
  • FIG. 1 shows, by way of illustrative and non -limitative example, a configuration for a real-time label superimposition system 100 according to the present invention.
  • the real -time label superimposition system 100 can be realized, for example, as a modified form of the digital camera in FIG. 1 of the above-mentioned Miyake patent.
  • the system 100 includes front-end optics 104, a view finder 108, a GPS unit 112, a liquid crystal display (LCD) monitor 116, an orientation sensor 120, a proces sing unit or processor 124, a user input device 128 and a world model 132, all connected on a data and control bus 136.
  • the front-end optics 104 may include a taking lens comprising a focus lens and a variable magnification lens, a diaphragm, and an optical low pass filter, these being arranged in series .
  • the view finder 108 has a front-end LCD 110, as in Miyake, but this LCD functions as a label superimposition unit.
  • the LCD 110 is preferably extended across the entire viewing lens of the view finder 108 to allow label superimposition anywhere in the image.
  • the GPS unit 112 is similar to that in Miyake, continuously providing updates as to ground coordinates and altitude.
  • LCD monitor 116 is optionally provided with the capability to display the superimposed labeling which has been embedded into the processing unit's video output to the LCD monitor.
  • the orientation sensor 120 detects camera tilt and horizontal rotation, as by the use of micro-machined silicon (“MEMS”) sensors . Such sensors are discussed in the Dunton patent. One sensor is preferably utilized to detect tilt and another to detect rotation. Alternatively, the tilt sensor in the Bryant patent of a conventional film camera can be implemented.
  • MEMS micro-machined silicon
  • Processing unit 124 may include the Miyake strobe unit, charge -coupled device
  • the Miyake switch unit may be integrated into the user input device 128 of the present invention. Along with a release switch for taking photos, the switch unit includes a user-operable zooming/magnification factor selector.
  • the user input device 128 preferably includes, in addition, a wheel, button or other mechanism for choosing among displayed labels in the image, and possibly other input means by which to filter, by classification, labels to be displayed.
  • the user can filter tourist information (e.g. this is the Eiffel Tower), topographical information (e.g. Paris with 10,000,000 inhabitants, and so on).
  • the user input device 128 may additionally feature a mechanism, such as a trackball, joystick or rocker switch, by which to navigate a cursor across the screen of the LCD monitor 116.
  • This user interface can be utilized, for example, to limit concurrent display of labels to points/objects of interest in the immediate vicinity of the curs or, so as to not overcrowd the image with labels.
  • World model 132 is a database, which may be incorporated into the Miyake PC card.
  • the database 132 includes points of interest and objects of interest 140, and optionally shapes 144 of the objects. Loading of the database 132 ahead of time is especially useful if the camera is integrated with a personal digital assistant (PDA).
  • PDA personal digital assistant
  • the database 132 may alternatively be accessed via a wireless connection to the system or camera 100 from an Internet service affording WiFi (IEEE 802.11 wireless Ethernet) communication. Such access is especially useful if the camera 100 is integrated with a mobile phone.
  • this "travel -guide" information can be loaded into another portable device such as a PDA, which can wirelessly connect to the camera 100 via a Body Area Network or Personal Area Network.
  • Processing itself may also be off-loaded to a remote server (not shown) to reduce form factor and power requirements for the mobile image-viewing device.
  • Updates from the GPS unit 112 and orientation sensor 120 can be wirelessly transmitted to the server.
  • the server determines whether objects/points of interest are within the field of view - and, if so, where.
  • the server transmits this determined data back to the camera 100 so that the processor 124 can appropriately operate the view finder 108 to superimpose labeling.
  • FIG. 2 portrays two exemplary views 204, 208 from a digital camera incorporating the real-time label superimposition system 100 of the present invention.
  • the left-hand view 204 shows a label 212 having a stem portion 216 and containing a description 220 of an object of interest, namely the Matterhorn mountain.
  • Other labels 224 are positioned near other respective objects or points of interest.
  • An example of a point of interest is a particular location within a nature trail that affords a desirable view of a canyon.
  • the user can switch to any of the other labels 224, thereby generating a descriptive label at the chosen location.
  • the right-hand view 208 displays a label 228 that identifies Mont Blanc.
  • the descriptive label may provide a number or other identifier that matches with a travel guide or hiking route.
  • the auto-labeling function could be editable by the user by means of the user input device 128.
  • the user could add information to labels or even create new labels such as "Home,” “starting point of the trip,” “my hotel room,” and so forth.
  • Non-descriptive labels only appear when more than one point/object of interest is concurrently displayed. Otherwise, if only a single point/object of interest is displayed, the only label appearing is the descriptive label. Alternatively, the user might override the default to allow more than one descriptive label, if applicable, to concurrently display.
  • the LCD 110 in the view finder 108 responsive to electrical signals, selectively obscures incident light to form the letters 232 "Mont Blanc" and the label outline 236.
  • the obscuring elements may be colored to form a colored label within the image 208.
  • other light sources such as light -emitting diodes (LEDs) may be used in combination panels that are transmissive in one direction and reflective in the other direction to introduce into a common light path both the original image and the superimposed label.
  • FIG. 3 conceptually depicts an example of a field of view in two dimensions in accordance with the present invention. Since the image 302 as shown in FIG.
  • FIG. 3 is rectangular, i.e., viewable through a rectangular viewing lens of the view finder 108, the field of view is defined simply by a horizontal sector angle 303 and a vertical sector angle 304, and, more generally, by the focal length of the taking lens in the front -end optics 104 and by the horizontal and vertical dimensions 306, 308 of the viewing lens of the view finder 108.
  • Such truncation of the image 302 to the dimensions 306, 308 may occur anywhere in the light path, starting at the front-end and continuing through the view finder 108, and is shown in FIG. 3 at a particular location merely for simplicity of demonstration.
  • the field of view is defined by exterior walls 312, 316, 320 of a solid figure, in this case a pyramid 324, whose apex coincides with the viewpoint 328, such that the walls extend through a periphery 332 of the image 302.
  • the apex or viewpoint 328 is the optical center of the taking lens in the front-end optics 104.
  • a focused rendition of the image formed at a focal length beyond the viewpoint 328 is routed to the view finder 108, where a label may be superimposed according to the invention. That rendition is also made incident upo n a front surface of the CCD within the processing unit 124.
  • variable magnification or "zoom" lens in the front -end optics 104 features multiple different focal lengths, each entailing a different field of view.
  • the field of view is defined in FIG. 3 by a pyramidal shape, implementation of the invention in, for example, binoculars would limit the field of view in accordance with a conical shape, e.g. to a single, uniform sector angle.
  • the field of view more generally, may be defined with respect to any arbitrary shape for the view finder 108.
  • the processing unit 124 can determine by means of known geometrical transformations which, if any, point/objects of interest 140 within the world model 132 are currently within the current field of view. For example, as shown in FIG. 3, an object 328 at world coordinates (X,Y,Z) is currently within the field of view and displayed in the view finder 108 as object image 332 at image or view finder coordinates (x,y). Accordingly, the label (not shown) stored in the world model 132 for object 328 is automatically placed in real -time adjacent to the object image 332 to identify the object.
  • the world model 132 includes for the objects/points of interest 140 topographical data so that the system 100 accounts for the possibility of occlusion of the object/point behind an object in the foreground.
  • Such modeling techniques are often utilized, for example, in on-screen portrayal of three-dimensional objects in computer graphics applications, such as computer animations. If the object/point is totally occluded, either the object/point is not labeled or, optionally, the stem on the label may be dotted to indicate occlusion.
  • FIG. 4 sets forth one example of a real-time label -superimposing process 400 for a point/object of interest within the viewed image in accordance with the present invention.
  • the device of the present invention may be incorporated into a camera, camcorder, binoculars or night- viewing equipment, for example. If the image -viewing apparatus is on (step S404), the viewpoint and orientation are read. Both of these parameters may be repeatedly updated synchronously or asynchronously and buffered for reading. In addition, the most -recently buffered zooming/magnification factor can be read, to determine the most recent output of the user-operable zooming/magnification factor selector (step S408).
  • This database search may, alternatively, be limited to points/objects within the FOV that are approximately at a particular distance from the viewpoint.
  • a focusing sensor is typically provided, within the camera unit of the processing unit 124, for calculating a distance from the camera 100 at which image focusing occurs. Labeling can accordingly be limited to only those objects/points within the FOV that are also at that distance or within an interval centered at that distance.
  • the location within the image 302 is determined for each point object within the FOV, optionally inclusive of merely those disposed approximately at the calculated focusing distance. This determined location is used to update the average location, if any yet exists, for that point/object.
  • An average location is desirable, to reduce noise which appears as random error in the readings from the orientation sensor 120 and the GPS unit 112. The randomness of the errors tends to make the errors offset each other when the readings are averaged.
  • the appropriate label is then superimposed onto the image 302 for viewing by the user (step S416). As more and more readings for a given object/point are averaged, jitter in the label is mitigated and stability is achieved.
  • step S420 is made by the processing as to whether the determined location for the object point, and correspondingly for the label, is stable, based for instance on a predetermined number of readings (step S420). If more readings are needed, the process repeats, starting at step S404. Preferably, the needed number of iterations is achieved for a single, point/object before processing a next point/object. If, on the other hand, stability has been achieved, a corresponding label is linked in fixed "on-screen" relation to tracked contours in the image 302 (step S424).
  • Tracking can be implemented using the same methodology conventionally applied in motion compensation, examining candidate images within a predetermined window in a subsequent frame for the best-matching image. Once the label is linked to the tracked contours, location averaging is no longer needed. The label stems continues to p oint to the object point even if the camera 100 is moved by the user, as long as the point/object remains within the field of view. A separate process (not shown) can monitor linked labels, and drop the link whenever the point/object exits the field of vi ew.
  • FIG. 4 represents merely an example of the specific design.
  • the query as to location stability may, in addition to determining whether the average has been computed with the predetermined number of summand, also take into account detection by the MEMS sensors of recent significant camera movement.
  • the repeated updating of parameters may include retention of respective rolling averages for the viewpoint and orientation, the rolling average having a set number of summands.
  • the query as to location stability in step S420 would then be reduced to identifying when stability of the rolling average had not yet been achieved, e.g., when the camera has just been turned on or when the MEMS sensors detect re cent significant movement in the camera.
  • FIG. 5 illustrates a stored -shape mapping sub -process 500 for mapping to the viewed image a shape 504 stored among the object shapes 144 in the world model 132 in accordance with the present invention.
  • the sub-process 500 reduces systematic error in the orientation and GPS readings, and may be executed between steps S420 and S424 of the real-time label- superimposing process 400.
  • the shape 504 of the object of interest 508, in this case th e "Toronto Tower,” is, as shown in FIG. 6, first mapped to the average location determined in step S416 once the location has been determined to be stable in step S420. Then, a process similar to that described above for contour tracking may be performed. Examination is made within a search window of the mapped image for a best match, the search window being of predetermined size (step S608). The average location is then updated by the translation determined to align the mapped image with the best matchi ng candidate image (step S612). The corresponding label placed to point to the updated location is then linked in step S424, thereby reducing systematic error.
  • a preferred embodiment not only minimizes random and systematic error in the parameter readings, but reduces the storage/processing overhead by limiting the world model database 132 to data specific to the objects/points of interest.
  • the database 132 need not include data throughout three-dimensional space to enable determination as to whether occlusion of the point/object exists. It suffices to model merely the points/objects of interest to include characteristics such as, for example, shape, color and shading, these being stored for multiple possible orientations or alternatively i n real-time extracted from a three- dimensional model. This expands upon the concept introduced in FIG. 6 of storing shape information. As in the FIG.
  • step S416 query is made as to whether the point/object is visible in the image 302.
  • This query involves retrieving the stored characteristics for the point/object and comparing them within a window of the image 302, that window being centered at the average location in a manner similar to that of step S608.
  • the comparison may involve, for instance, merely a contour based on the retrieved characteristics as matched against a detected contour within the window.
  • the image characteristics to be ma tched need not be exactly co-located with the point/object, but may, instead, lie within a vicinity of the point/object.
  • the object/point is judged to be not visible in the image. Otherwise, if the match is close enough, the average location is updated as in step S612, not only reducing systematic error, but, in addition, accounting for occlusion or non-occlusion of the object/point.
  • This approach may be further modified, as mentioned above, by retention of respective rolling averages for the viewpoint and orientation.
  • the extra storage and processing overhead for the characteristic data may suggest off-loading, from a mobile device, the world model and much of the image processing to the aforementioned server.
  • the camera/binoculars 100 may transmit the averaged parameter readings to the server (step S408).
  • the server determines potential points/objects of interest that are included within the FOV, not taking into account occlusion. For each point/object, it determines how the object would look in the image 302, based on the received parameter readings. At least partial images are transmitted back including approximately where in the image 302 they are located. The above -described pixel-to-pixel matching is then applied.
  • the images transmitted back can be limited to particular camera-requested categories.
  • Video captured by a camcorder for example, can be downloaded to a personal computer (PC) that efficiently identifies points/objects of interest while playing back the video.
  • PC personal computer
  • the image-viewing apparatus such as a camcorder, retains, in storage, contours that are linked to a label in step S424 (step S704).
  • the stored video and contours are downloaded to the PC (step S708), or may be physically transferred as by means of a PC card.
  • the contour(s) are matched to the playback image to identify the corresponding points/objects of interest.
  • identification also makes available for presentation any metadata that has been stored in connection with the contour, such as the descriptive label or an identification based on GPS and a mapping database as to the town or other location of the user at the time the video was recorded.
  • the labeling in the video playback may therefore first occur for an image recorded earlier than the image for which the labeling first appeared in the view finder of the camcorder.
  • the contour retention and matching also applies to the digital photos stored for a digital camera such as that described with respect to FIG. 1.
  • the contour(s) for visible points/objects of interest, and associated metadata are retained for subsequent matching of the contours to photos displayed on the PC.
  • the metadata is displayed on-screen, superimposed on or alongside the respective photo. In this way, vacation photos are much easier to relive.
  • FIG. 8 even the need to rematch contours can be eliminated, by storing the updated label location in the frame every time the video or photo frame is stored.
  • the label is linked, with respect to location, in fixed relation to the currently tracked contour(s) (step S424), and since the image coordinates of the currently tracked contour(s) are known, the image coordinates of the label are known and this information, along with any description, may be embedded in the frame to be stored.
  • the stored photos or video are replayed on the PC, these stored coordinates are read and used to correctly superimpose the stored labels.
  • any other metadata stored with the frame can be accessed for visual display, aural presentation as by means of a speech synt hesizer, etc.
  • Query is first made as to whether a video or photo frame is to be stored (step S804).
  • the label location which may have been updated (step S612) and is linked in fixed relation to the contour(s) (step S424) is stored along with the frame to be stored.
  • an identifier of the label location, and any metadata may be stored as part of the frame or in linked relation to the frame (steps S808, S812).
  • the PC superimposes the label based on the identifier and performs any further presentation based on the metadata.
  • This playback technique may also be applied, for example, to superimpose labeling in the LCD monitor 116 concurrently with image/video capture, in which case, the frame, label location(s) and description are stored in a buffer.
  • a user -friendly adjunct to a digital camera, binoculars or other image-viewing apparatus automatically provides in real time labels to identify known points/objects of interest.
  • An enhanced image -viewing experience is afforded, particularly for the tourist.
  • the user may be provided with an actuator on the user input device 128 by which to introduce onto or exclude from the stored digital photos labels displayed in the view finder. It is therefore intended that the invention be not limited to the exact forms described and illustrated, but should be constructed to cover all modifications that may fall within the scope of the appended claims.

Abstract

A device operable with an image-viewing apparatus identifies known points/objects of interest in an image and superimposes onto the image being viewed labels to identify the known points/objects. Identification is made based on stored information for each point/object in a world model database and based on device location and orientation parameters that are updated automatically in real time.

Description

AUTOMATIC, REAL-TIME, SUPERIMPOSED LABELING OF POINTS AND OBJECTS OF INTEREST WITHIN A VIEW
The present invention relates to superimposing, onto an image, labels of points and objects appearing in the image. More particularly, the present invention relate s to automatically in real-time determining if a predetermined point or object of interest appears in an image and, if so, superimposing the corresponding label onto the image. It is known to determine location and orientation of a camera taking pictures. The registration of location for the purpose of easy browsing and retrieval of the pictures at a later time is described in U.S. Patent Nos. 5,506,644 to Suzuki et al. and 6,222,985 to Miyake
(hereinafter "Miyake"), the disclosure of the latter being hereby incorporated by reference in its entirety. It is also known to add direction/orientation information to pictures. For instance, U.S. Patent No. 5,614,981 to Bryant et al. (hereinafter "Bryant"), the disclosure of which is hereby fully incorporated by reference herein, proposes a camera with a magnetic orientation sensor. U.S. Patent 5,659,805 to Furlani et al. describes a camera for indicating camera orientations on photographic film, and U.S. Patent No. 5,699,444 to Palm describes how image data (known objects on the photograph of which the exact location is known) can be used to determine the orientation of the camera. U.S. Patent No. 4,183,645 to Ohmura et al. relates to detecting the orientation of the photo (portrait or landscape) in order to add the date and time in the appropriate orientation. As a further application regarding orientation, U.S. Patent No. 6,304,284 to Dunton et al. (hereinafter "Dunton"), whose disclosure is herein incorporated by reference in its entirety, uses orientation in formation to create panoramic images with a camera. Global positioning system (GPS) receivers which deliver real -time ground coordinate and altitude readings for the receiver can now be integrated into a single chip and are available to be built into all kinds of portable devices such as mobile phones and digital cameras, at low cost. There exists, furthermore, an observable trend of more and more digital photos and photo albums being shared on the Internet. What is needed is a way to utilize readily accessible location and orientation data real-time to enhance the experience of taking a photo. The present invention has been made to address the above -noted shortcomings in the prior art. It is an object of the invention to provide a device, a method operabl e on the device, and a computer program for performing the method, wherein a device usable with an image - viewing apparatus marks, automatically in real -time, an image captured from a viewpoint. The image is bounded in two dimensions by a field of view at a particular orientation. The device communicates with a storage medium for storing location information of a point of interest and/or an object of interest. The device includes a processing unit configured to, automatically in real-time, update a value of at least one parameter that defines the capturing of the image from the viewpoint. The processor also determines, based on the location information and at least one updated value, whether the respective point or object is disposed within the field of view at said viewpoint at said particular orientation. Further included in the device is a label superimposing unit configured to, automatically in real -time, based on the determination that the respective point or object is so disposed, and under control of the processing unit, superimpose a label onto the image to identify the respective point or object. Details of the invention disclosed herein shall be described with the aid of the figures listed below, wherein: FIG. 1 is a schematic diagram of a real-time label superimposition system in accordance with the present invention; FIG. 2 is a how an image with superimposed labels appears to the user through the viewfinder of a camera in accordance with the present invention; FIG. 3 is a conceptual diagram of a two-dimensional field of view in accordance with the present invention; FIG. 4 is a flowchart of a process for superimposing a label for a point/object of interest within the viewed image in accordance with the present invention; FIG. 5 is depiction of a sub-process of mapping a stored shape to the viewed image in accordance with the present invention; FIG. 6 is a flowchart of a sub -process for refining label placement in accordance with the present invention using the sub-process depicted in FIG. 5; FIG. 7 is a flowchart of a method for retrieving labels and associated metadata to be displayed upon playback or re -presentation of recorded images; and FIG. 8 is a flowchart of a process for retaining label information for subsequent representation of the stored images. FIG. 1 shows, by way of illustrative and non -limitative example, a configuration for a real-time label superimposition system 100 according to the present invention. The real -time label superimposition system 100 can be realized, for example, as a modified form of the digital camera in FIG. 1 of the above-mentioned Miyake patent. The system 100 includes front-end optics 104, a view finder 108, a GPS unit 112, a liquid crystal display (LCD) monitor 116, an orientation sensor 120, a proces sing unit or processor 124, a user input device 128 and a world model 132, all connected on a data and control bus 136. As in Miyake, the front-end optics 104 may include a taking lens comprising a focus lens and a variable magnification lens, a diaphragm, and an optical low pass filter, these being arranged in series . The view finder 108 has a front-end LCD 110, as in Miyake, but this LCD functions as a label superimposition unit. As such, the LCD 110 is preferably extended across the entire viewing lens of the view finder 108 to allow label superimposition anywhere in the image. The GPS unit 112 is similar to that in Miyake, continuously providing updates as to ground coordinates and altitude. LCD monitor 116 is optionally provided with the capability to display the superimposed labeling which has been embedded into the processing unit's video output to the LCD monitor. The orientation sensor 120 detects camera tilt and horizontal rotation, as by the use of micro-machined silicon ("MEMS") sensors . Such sensors are discussed in the Dunton patent. One sensor is preferably utilized to detect tilt and another to detect rotation. Alternatively, the tilt sensor in the Bryant patent of a conventional film camera can be implemented. It is noted here that, in addition to orientation, the zooming/magnification factor of the variable magnification lens in the front -end optics 104 must be known to determine whether a point of interest or object of interest is currently within the field of view of the camera 100, as discussed below in further detail. Processing unit 124 may include the Miyake strobe unit, charge -coupled device
(CCD) unit, image pickup unit, camera unit, display unit, process unit and output unit. Accordingly, the image emitted from the low -pass filter in the front-end optics 104, as well as being displayed in the view finder 108 after label superimposition, is incident upon the CCD unit within the processing unit 124. The latter is necessary for storage of the image as a digital photo and for display on the LCD monitor 116. The Miyake switch unit may be integrated into the user input device 128 of the present invention. Along with a release switch for taking photos, the switch unit includes a user-operable zooming/magnification factor selector. The user input device 128 preferably includes, in addition, a wheel, button or other mechanism for choosing among displayed labels in the image, and possibly other input means by which to filter, by classification, labels to be displayed. As an example of the latter, the user can filter tourist information (e.g. this is the Eiffel Tower), topographical information (e.g. Paris with 10,000,000 inhabitants, and so on). The user input device 128 may additionally feature a mechanism, such as a trackball, joystick or rocker switch, by which to navigate a cursor across the screen of the LCD monitor 116. This user interface can be utilized, for example, to limit concurrent display of labels to points/objects of interest in the immediate vicinity of the curs or, so as to not overcrowd the image with labels. World model 132 is a database, which may be incorporated into the Miyake PC card. The database 132 includes points of interest and objects of interest 140, and optionally shapes 144 of the objects. Loading of the database 132 ahead of time is especially useful if the camera is integrated with a personal digital assistant (PDA). The database 132 may alternatively be accessed via a wireless connection to the system or camera 100 from an Internet service affording WiFi (IEEE 802.11 wireless Ethernet) communication. Such access is especially useful if the camera 100 is integrated with a mobile phone. As a further alternative, this "travel -guide" information can be loaded into another portable device such as a PDA, which can wirelessly connect to the camera 100 via a Body Area Network or Personal Area Network. Processing itself may also be off-loaded to a remote server (not shown) to reduce form factor and power requirements for the mobile image-viewing device. Updates from the GPS unit 112 and orientation sensor 120, for example, can be wirelessly transmitted to the server. The server determines whether objects/points of interest are within the field of view - and, if so, where. The server transmits this determined data back to the camera 100 so that the processor 124 can appropriately operate the view finder 108 to superimpose labeling. FIG. 2 portrays two exemplary views 204, 208 from a digital camera incorporating the real-time label superimposition system 100 of the present invention. The left-hand view 204 shows a label 212 having a stem portion 216 and containing a description 220 of an object of interest, namely the Matterhorn mountain. Other labels 224 are positioned near other respective objects or points of interest. An example of a point of interest is a particular location within a nature trail that affords a desirable view of a canyon. By manipulating the user input device 128, the user can switch to any of the other labels 224, thereby generating a descriptive label at the chosen location. Accordingly, the right-hand view 208 displays a label 228 that identifies Mont Blanc. In the case of the nature trail, for instance, the descriptive label may provide a number or other identifier that matches with a travel guide or hiking route. The auto-labeling function could be editable by the user by means of the user input device 128. Thus, the user could add information to labels or even create new labels such as "Home," "starting point of the trip," "my hotel room," and so forth. Non-descriptive labels only appear when more than one point/object of interest is concurrently displayed. Otherwise, if only a single point/object of interest is displayed, the only label appearing is the descriptive label. Alternatively, the user might override the default to allow more than one descriptive label, if applicable, to concurrently display. The LCD 110 in the view finder 108, responsive to electrical signals, selectively obscures incident light to form the letters 232 "Mont Blanc" and the label outline 236. Optionally, the obscuring elements may be colored to form a colored label within the image 208. As an alternative to the use of LCDs, other light sources such as light -emitting diodes (LEDs) may be used in combination panels that are transmissive in one direction and reflective in the other direction to introduce into a common light path both the original image and the superimposed label. FIG. 3 conceptually depicts an example of a field of view in two dimensions in accordance with the present invention. Since the image 302 as shown in FIG. 3 is rectangular, i.e., viewable through a rectangular viewing lens of the view finder 108, the field of view is defined simply by a horizontal sector angle 303 and a vertical sector angle 304, and, more generally, by the focal length of the taking lens in the front -end optics 104 and by the horizontal and vertical dimensions 306, 308 of the viewing lens of the view finder 108. Such truncation of the image 302 to the dimensions 306, 308 may occur anywhere in the light path, starting at the front-end and continuing through the view finder 108, and is shown in FIG. 3 at a particular location merely for simplicity of demonstration. In effect, the field of view is defined by exterior walls 312, 316, 320 of a solid figure, in this case a pyramid 324, whose apex coincides with the viewpoint 328, such that the walls extend through a periphery 332 of the image 302. The apex or viewpoint 328 is the optical center of the taking lens in the front-end optics 104. A focused rendition of the image formed at a focal length beyond the viewpoint 328 is routed to the view finder 108, where a label may be superimposed according to the invention. That rendition is also made incident upo n a front surface of the CCD within the processing unit 124. The variable magnification or "zoom" lens in the front -end optics 104 features multiple different focal lengths, each entailing a different field of view. Although the field of view is defined in FIG. 3 by a pyramidal shape, implementation of the invention in, for example, binoculars would limit the field of view in accordance with a conical shape, e.g. to a single, uniform sector angle. The field of view, more generally, may be defined with respect to any arbitrary shape for the view finder 108. Based on current ground coordinates and elevation provided by the GPS unit 112, tilt and rotational angle provided by the orientation sensor 120 and represented by the arrow 336, the processing unit 124 can determine by means of known geometrical transformations which, if any, point/objects of interest 140 within the world model 132 are currently within the current field of view. For example, as shown in FIG. 3, an object 328 at world coordinates (X,Y,Z) is currently within the field of view and displayed in the view finder 108 as object image 332 at image or view finder coordinates (x,y). Accordingly, the label (not shown) stored in the world model 132 for object 328 is automatically placed in real -time adjacent to the object image 332 to identify the object. Preferably, the world model 132 includes for the objects/points of interest 140 topographical data so that the system 100 accounts for the possibility of occlusion of the object/point behind an object in the foreground. Such modeling techniques are often utilized, for example, in on-screen portrayal of three-dimensional objects in computer graphics applications, such as computer animations. If the object/point is totally occluded, either the object/point is not labeled or, optionally, the stem on the label may be dotted to indicate occlusion. FIG. 4 sets forth one example of a real-time label -superimposing process 400 for a point/object of interest within the viewed image in accordance with the present invention. The device of the present invention may be incorporated into a camera, camcorder, binoculars or night- viewing equipment, for example. If the image -viewing apparatus is on (step S404), the viewpoint and orientation are read. Both of these parameters may be repeatedly updated synchronously or asynchronously and buffered for reading. In addition, the most -recently buffered zooming/magnification factor can be read, to determine the most recent output of the user-operable zooming/magnification factor selector (step S408). For each point/object of interest in the world model 132 that has not already been linked to a label (as will be explained), a decision is made as to whether the point/object is within the current field of view (FOV) at the current viewpoint at the current orientation (step S412). This database search may, alternatively, be limited to points/objects within the FOV that are approximately at a particular distance from the viewpoint. Thus, as in Miyake, a focusing sensor is typically provided, within the camera unit of the processing unit 124, for calculating a distance from the camera 100 at which image focusing occurs. Labeling can accordingly be limited to only those objects/points within the FOV that are also at that distance or within an interval centered at that distance. In any case, , the location within the image 302 is determined for each point object within the FOV, optionally inclusive of merely those disposed approximately at the calculated focusing distance. This determined location is used to update the average location, if any yet exists, for that point/object. An average location is desirable, to reduce noise which appears as random error in the readings from the orientation sensor 120 and the GPS unit 112. The randomness of the errors tends to make the errors offset each other when the readings are averaged. The appropriate label is then superimposed onto the image 302 for viewing by the user (step S416). As more and more readings for a given object/point are averaged, jitter in the label is mitigated and stability is achieved. When people take pictures, for example, it can easily take 5-10 seconds to find the right composition and settings for the camera, whereas multiple readings for an average can be accumulated quickly, e.g., within a fraction of a second. Query is made by the processing as to whether the determined location for the object point, and correspondingly for the label, is stable, based for instance on a predetermined number of readings (step S420). If more readings are needed, the process repeats, starting at step S404. Preferably, the needed number of iterations is achieved for a single, point/object before processing a next point/object. If, on the other hand, stability has been achieved, a corresponding label is linked in fixed "on-screen" relation to tracked contours in the image 302 (step S424). Techniques in detecting contours in an image are well-known. Tracking can be implemented using the same methodology conventionally applied in motion compensation, examining candidate images within a predetermined window in a subsequent frame for the best-matching image. Once the label is linked to the tracked contours, location averaging is no longer needed. The label stems continues to p oint to the object point even if the camera 100 is moved by the user, as long as the point/object remains within the field of view. A separate process (not shown) can monitor linked labels, and drop the link whenever the point/object exits the field of vi ew. FIG. 4 represents merely an example of the specific design. Thus, for example, the query as to location stability (step S420) may, in addition to determining whether the average has been computed with the predetermined number of summand, also take into account detection by the MEMS sensors of recent significant camera movement. The repeated updating of parameters, as a further example, may include retention of respective rolling averages for the viewpoint and orientation, the rolling average having a set number of summands. The query as to location stability in step S420 would then be reduced to identifying when stability of the rolling average had not yet been achieved, e.g., when the camera has just been turned on or when the MEMS sensors detect re cent significant movement in the camera. In addition, the query as to existence of non -linked points/objects with the FOV in step S412 may, in a single iteration, determine all such points/objects. Likewise, superimposition of labels in step S41 , and label linking in step S424, would be performed in a single iteration. FIG. 5 illustrates a stored -shape mapping sub -process 500 for mapping to the viewed image a shape 504 stored among the object shapes 144 in the world model 132 in accordance with the present invention. The sub-process 500 reduces systematic error in the orientation and GPS readings, and may be executed between steps S420 and S424 of the real-time label- superimposing process 400. The shape 504 of the object of interest 508, in this case th e "Toronto Tower," is, as shown in FIG. 6, first mapped to the average location determined in step S416 once the location has been determined to be stable in step S420. Then, a process similar to that described above for contour tracking may be performed. Examination is made within a search window of the mapped image for a best match, the search window being of predetermined size (step S608). The average location is then updated by the translation determined to align the mapped image with the best matchi ng candidate image (step S612). The corresponding label placed to point to the updated location is then linked in step S424, thereby reducing systematic error. A preferred embodiment not only minimizes random and systematic error in the parameter readings, but reduces the storage/processing overhead by limiting the world model database 132 to data specific to the objects/points of interest. In particular, the database 132 need not include data throughout three-dimensional space to enable determination as to whether occlusion of the point/object exists. It suffices to model merely the points/objects of interest to include characteristics such as, for example, shape, color and shading, these being stored for multiple possible orientations or alternatively i n real-time extracted from a three- dimensional model. This expands upon the concept introduced in FIG. 6 of storing shape information. As in the FIG. 6 embodiment, once the object/point is determined to be within the FOV, known image recognition techniques can be employed to decide whether the object/point is sufficiently occluded so as to warrant dotted labeling or no labeling at all, depending on the implementation. More specifically, after execution of the location averaging sub-step (in step S416), query is made as to whether the point/object is visible in the image 302. This query involves retrieving the stored characteristics for the point/object and comparing them within a window of the image 302, that window being centered at the average location in a manner similar to that of step S608. The comparison may involve, for instance, merely a contour based on the retrieved characteristics as matched against a detected contour within the window. It is also noted that the image characteristics to be ma tched need not be exactly co-located with the point/object, but may, instead, lie within a vicinity of the point/object. In any event, if a predetermined level of pixel-to-pixel matching is not attained, the object/point is judged to be not visible in the image. Otherwise, if the match is close enough, the average location is updated as in step S612, not only reducing systematic error, but, in addition, accounting for occlusion or non-occlusion of the object/point. This approach may be further modified, as mentioned above, by retention of respective rolling averages for the viewpoint and orientation. Despite paring down the world model 132 by the above -described method, the extra storage and processing overhead for the characteristic data may suggest off-loading, from a mobile device, the world model and much of the image processing to the aforementioned server. With respect to the image processing, the camera/binoculars 100 may transmit the averaged parameter readings to the server (step S408). The server determines potential points/objects of interest that are included within the FOV, not taking into account occlusion. For each point/object, it determines how the object would look in the image 302, based on the received parameter readings. At least partial images are transmitted back including approximately where in the image 302 they are located. The above -described pixel-to-pixel matching is then applied. To prevent a wireless -bandwidth bottleneck in the event of multiple currently- viewed points/objects of interest, the images transmitted back can be limited to particular camera-requested categories. Video captured by a camcorder, for example, can be downloaded to a personal computer (PC) that efficiently identifies points/objects of interest while playing back the video. As shown in FIG. 7, the image-viewing apparatus, such as a camcorder, retains, in storage, contours that are linked to a label in step S424 (step S704). The stored video and contours are downloaded to the PC (step S708), or may be physically transferred as by means of a PC card. As the video plays back, the contour(s) are matched to the playback image to identify the corresponding points/objects of interest. Such identification also makes available for presentation any metadata that has been stored in connection with the contour, such as the descriptive label or an identification based on GPS and a mapping database as to the town or other location of the user at the time the video was recorded. Although a delay may exist between the time a contour or set of contours first appears on the video and the time at which the real-time label superimposition system 100 "locks" onto the contour(s) in step S424, the matching by the PC on playback identifies the object point as soon as the appropriate contour(s) appear. Advantageously, the labeling in the video playback may therefore first occur for an image recorded earlier than the image for which the labeling first appeared in the view finder of the camcorder. The contour retention and matching also applies to the digital photos stored for a digital camera such as that described with respect to FIG. 1. As a photo is taken, the contour(s) for visible points/objects of interest, and associated metadata, are retained for subsequent matching of the contours to photos displayed on the PC. Once a match is made, the metadata is displayed on-screen, superimposed on or alongside the respective photo. In this way, vacation photos are much easier to relive. Referring to FIG. 8, even the need to rematch contours can be eliminated, by storing the updated label location in the frame every time the video or photo frame is stored. In particular, since the label is linked, with respect to location, in fixed relation to the currently tracked contour(s) (step S424), and since the image coordinates of the currently tracked contour(s) are known, the image coordinates of the label are known and this information, along with any description, may be embedded in the frame to be stored. When the stored photos or video are replayed on the PC, these stored coordinates are read and used to correctly superimpose the stored labels. Simultaneously, any other metadata stored with the frame can be accessed for visual display, aural presentation as by means of a speech synt hesizer, etc. Query is first made as to whether a video or photo frame is to be stored (step S804). If so, the label location which may have been updated (step S612) and is linked in fixed relation to the contour(s) (step S424) is stored along with the frame to be stored. In particular, an identifier of the label location, and any metadata, may be stored as part of the frame or in linked relation to the frame (steps S808, S812). During re -presentation or playback of the frame on the PC, for example, the PC superimposes the label based on the identifier and performs any further presentation based on the metadata. This playback technique may also be applied, for example, to superimpose labeling in the LCD monitor 116 concurrently with image/video capture, in which case, the frame, label location(s) and description are stored in a buffer. As has been demonstrated above, a user -friendly adjunct to a digital camera, binoculars or other image-viewing apparatus automatically provides in real time labels to identify known points/objects of interest. An enhanced image -viewing experience is afforded, particularly for the tourist. While there have been shown and described what are considered to be preferred embodiments of the invention, it will, of course, be understood that various modifications and changes in form or detail could readily be made without departing from the spirit of the invention. For example, the user may be provided with an actuator on the user input device 128 by which to introduce onto or exclude from the stored digital photos labels displayed in the view finder. It is therefore intended that the invention be not limited to the exact forms described and illustrated, but should be constructed to cover all modifications that may fall within the scope of the appended claims.

Claims

WHAT IS CLAIMED IS:
1. A device (100) usable with an image-viewing apparatus (S404), said device for marking, automatically in real-time, an image (302) captured from a viewpoint (328), said image being bounded in two dimensions by a field of view (303, 304) at a particular orientation (336), said device in communicative connection with a storage medium (132) for storing location information of at least one of a point of interest and an object of interest (140), said device comprising: a processor (124) configured to, automatically in real-time, update (S408) a value of at least one parameter that defines the capturing of said image from said viewpoint and determine, based on said location information and one or more of the at least one updated value, whether the respective point or object is disposed within said field of view at said viewpoint at said particular orientation (S412); and a label superimposing unit (110) configured to, automatically in real -time, based on the determination that the respective point or object is so disposed, and under control of said processor, superimpose upon said image a label to identify said respective point or object (S416).
2. The device of claim 1, wherein said one or more includes a va lue of at least one of said viewpoint and said particular orientation (S408).
3. The device of claim 1, wherein said one or more includes a value of a zooming/magnification factor (S408).
4. The device of claim 1, wherein said field of view is defined by exterior walls (312, 316, 320) of a solid figure (324) whose apex coincides with said viewpoint, said walls extending through a periphery (332) of said image.
5. The device of claim 1, wherein the superimposing includes: calculating where, in said image, said respective point or object is located based on said location information and one or more of the at least one updated value (S416); and arranging the label in said image based on the calculated location (S416).
6. The device of claim 5, wherein said updating is performed iteratively in real-time to generate a plurality of values for viewpoints, orientations or both (S408), so that the superimposing is performed based on an average location for the label (S416) and entails tracking movement of image content and linking said average location to the tracked content such that the label remains disposed in fixed relation to said content during the movement of said content (S424).
7. The device of claim 6, wherein the superimposing entails detecting a co ntour (S424), said image content comprising said contour.
8. The device of claim 1, wherein the storing step stores an object of interest (140).
9. The device of claim 8, wherein said location information includes a shape (144) of the stored object of interest and the superimposing entails: calculating where in said image said object is located (S416); and arranging the label in said image based on the calculated location (S416), said arranging involving: based on the calculated location, mapping the stored shape to said image (S604); and aligning the mapped shape to a shape in the image within a predetermined distance of the mapped shape (S608).
10. The device of claim 1, wherein the storing stores a plurality of points and/or objects of interest, the determining and superimposing being performed automatically in realtime for the plural points and/or objects (S412).
11. The device of claim 10, further including a user input device (128) for selecting among labels of respective ones of said plurality superimposed in said image for concurrent presentation (224), the currently selected label (220) presenting information descriptive of the respective point or object.
12. The device of claim 1, further including said image -viewing apparatus into which said device is integrated, said image -viewing apparatus including at least one of a camera and a pair of binoculars (S404).
13. A system comprising the integrated device of claim 12, said system further comprising said storage medium for storing location information for at least one of a point of interest and an object of interest, wherein said processor is further configured to read the stored location information and to control the label superimposing unit based on said information (112, 120, S412, S416), said device being further configured to present for viewing by a user said image containing the superimposed label (108, 116).
14. The device of claim 1, wherein the label has a portion for pointing to said respective point or object (216).
15. A method for marking an image automatically in real-time, said method comprising the steps of: storing, in a storage medium, location information for at least one of a point of interest and an object of interest (140); and providing an apparatus (100) that automatically in real-time performs the steps of: capturing (104), from a viewpoint (328), an image (302) bounded in two dimensions by a field of view (303, 304) at a particular orientation (336); updating a value of at least one parameter that defines the capturi ng of said image (S408); determining, based on said location information and one or more of the at least one updated value, whether the respective point or object is disposed within said field of view at said viewpoint at said particular orientation (S412); and based on the determination that said respective point or object is so disposed, superimposing upon said image a label to identify said respective point or object (S416).
16. The method of claim 15, wherein said one or more includes a value of at least one of said viewpoint, said particular orientation and a zooming/magnification factor (S408).
17. The method of claim 15, wherein the superimposing includes the steps of: calculating where in said image said respective point or object is located based on said location information and one or more of the at least one updated value (S416); and arranging the label in said image based on the calculated location (S416).
18. The method of claim 17, wherein the identifying is performed iteratively in realtime to generate a plurality of viewpoints, orientations or both (S412), so that the superimposing step comprises the step of deriving an average location for the label (S416), the superimposing step further comprising the steps of: tracking movement of image content (S424); and linking said average location to the tracked content such that the label remains disposed in fixed relation to said content during the movement of said content (S424).
19. The method of claim 15, wherein the storing step stores an object of interest, said location information includes a shape (144) of the stored object of interest and the superimposing step comprises the steps of: calculating where in said image said object is located (S416); and arranging the label in said image based on the calculated location, said arranging step further including the steps of: based on the calculated location, mapping the stored shape to said image (S604); and aligning the mapped shape to a shape in the image within a predetermined distance of the mapped shape (S608).
20. A computer-readable medium for storing a program (124) that includes instructions executable by a processor in performing automatically in real -time the steps of: capturing (104), from a viewpoint (328), an image (302) bounde d in two dimensions by a field of view (303, 304) at a particular orientation (336); updating a value of at least one parameter that defines the capturing of said image (S408); determining, based on location information for at least one of a point of interest and an object of interest and based on one or more of the at least one updated value, whether the respective point or object is disposed within said field of view at said viewpoint at said particular orientation (S412); and based on the determination that said respective point or object is so disposed, superimposing upon said image a label to identify said respective point or object (S416).
PCT/IB2005/051924 2004-06-16 2005-06-10 Automatic, real-time, superimposed labeling of points and objects of interest within a view WO2005124594A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US58015904P 2004-06-16 2004-06-16
US60/580,159 2004-06-16

Publications (1)

Publication Number Publication Date
WO2005124594A1 true WO2005124594A1 (en) 2005-12-29

Family

ID=34970075

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2005/051924 WO2005124594A1 (en) 2004-06-16 2005-06-10 Automatic, real-time, superimposed labeling of points and objects of interest within a view

Country Status (1)

Country Link
WO (1) WO2005124594A1 (en)

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006001129A1 (en) 2004-06-29 2006-01-05 Sony Corporation Information processing device and method, program, and information processing system
EP1840834A2 (en) 2006-03-27 2007-10-03 General Electric Company Article inspection apparatus
WO2009020785A1 (en) 2007-08-07 2009-02-12 Palm, Inc. Displaying image data and geographic element data
WO2009044343A2 (en) * 2007-10-05 2009-04-09 Nokia Corporation Method, apparatus and computer program product for multiple buffering for search application
EP2145157A1 (en) * 2007-05-14 2010-01-20 Robert Bosch GmbH Method for displaying video images and video system
EP2225607A2 (en) * 2007-12-24 2010-09-08 Microsoft Corporation Guided photography based on image capturing device rendered user recommendations
US20110199479A1 (en) * 2010-02-12 2011-08-18 Apple Inc. Augmented reality maps
WO2011156792A1 (en) * 2010-06-10 2011-12-15 Qualcomm Incorporated Acquisition of navigation assistance information for a mobile station
US8301159B2 (en) 2004-12-31 2012-10-30 Nokia Corporation Displaying network objects in mobile devices based on geolocation
US8310533B2 (en) 2006-03-27 2012-11-13 GE Sensing & Inspection Technologies, LP Inspection apparatus for inspecting articles
EP2365283A3 (en) * 2010-03-10 2013-01-23 Astrium GmbH Information output device
EP2610871A1 (en) * 2011-12-30 2013-07-03 Research In Motion Limited Methods and apparatus for use in providing object indicators during video recording or playback
EP2767953A1 (en) * 2013-02-13 2014-08-20 BlackBerry Limited Device with enhanced augmented reality functionality
US8855929B2 (en) 2010-01-18 2014-10-07 Qualcomm Incorporated Using object to align and calibrate inertial navigation system
CN104221056A (en) * 2010-02-05 2014-12-17 英特尔公司 Method for providing information on object within view of terminal device, terminal device for same and computer-readable recording medium
US20150022555A1 (en) * 2011-07-20 2015-01-22 Google Inc. Optimization of Label Placements in Street Level Images
US9208583B2 (en) 2013-02-13 2015-12-08 Blackberry Limited Device with enhanced augmented reality functionality
US9329052B2 (en) 2007-08-07 2016-05-03 Qualcomm Incorporated Displaying image data and geographic element data
US9519814B2 (en) 2009-06-12 2016-12-13 Hand Held Products, Inc. Portable data terminal
EP2385500A3 (en) * 2010-05-06 2017-07-05 LG Electronics Inc. Mobile terminal capable of providing multiplayer game and operating method thereof
US10445772B1 (en) 2011-09-09 2019-10-15 Google Llc Label placement based on objects in photographic images
US10942964B2 (en) 2009-02-02 2021-03-09 Hand Held Products, Inc. Apparatus and method of embedding meta-data in a captured image
US11170741B2 (en) 2012-02-29 2021-11-09 Nokia Technologies Oy Method and apparatus for rendering items in a user interface
GB2615628A (en) * 2021-12-07 2023-08-16 Motorola Mobility Llc Electronic device that presents location-tethered augmented reality notifications

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5699444A (en) * 1995-03-31 1997-12-16 Synthonics Incorporated Methods and apparatus for using image data to determine camera location and orientation
WO2001001075A2 (en) * 1999-06-25 2001-01-04 Bethere Photogrammetry engine for model construction
US6222985B1 (en) * 1997-01-27 2001-04-24 Fuji Photo Film Co., Ltd. Camera which records positional data of GPS unit
US6282362B1 (en) * 1995-11-07 2001-08-28 Trimble Navigation Limited Geographical position/image digital recording and display system
US20020052684A1 (en) * 1997-06-03 2002-05-02 Stephen Bide Portable information-providing apparatus
US20020075282A1 (en) * 1997-09-05 2002-06-20 Martin Vetterli Automated annotation of a view

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5699444A (en) * 1995-03-31 1997-12-16 Synthonics Incorporated Methods and apparatus for using image data to determine camera location and orientation
US6282362B1 (en) * 1995-11-07 2001-08-28 Trimble Navigation Limited Geographical position/image digital recording and display system
US6222985B1 (en) * 1997-01-27 2001-04-24 Fuji Photo Film Co., Ltd. Camera which records positional data of GPS unit
US20020052684A1 (en) * 1997-06-03 2002-05-02 Stephen Bide Portable information-providing apparatus
US20020075282A1 (en) * 1997-09-05 2002-06-20 Martin Vetterli Automated annotation of a view
WO2001001075A2 (en) * 1999-06-25 2001-01-04 Bethere Photogrammetry engine for model construction

Cited By (49)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1762822A4 (en) * 2004-06-29 2011-06-29 Sony Corp Information processing device and method, program, and information processing system
EP1762822A1 (en) * 2004-06-29 2007-03-14 Sony Corporation Information processing device and method, program, and information processing system
WO2006001129A1 (en) 2004-06-29 2006-01-05 Sony Corporation Information processing device and method, program, and information processing system
US8301159B2 (en) 2004-12-31 2012-10-30 Nokia Corporation Displaying network objects in mobile devices based on geolocation
CN103353726A (en) * 2006-03-27 2013-10-16 通用电气检查技术有限合伙人公司 Article inspection apparatus
EP1847960A2 (en) * 2006-03-27 2007-10-24 General Electric Company Inspection apparatus for inspecting articles
EP1840834A2 (en) 2006-03-27 2007-10-03 General Electric Company Article inspection apparatus
US8368749B2 (en) 2006-03-27 2013-02-05 Ge Inspection Technologies Lp Article inspection apparatus
CN101267546B (en) * 2006-03-27 2013-05-29 通用电气检查技术有限合伙人公司 Inspection apparatus for inspecting articles
EP1840834A3 (en) * 2006-03-27 2008-01-23 General Electric Company Article inspection apparatus
EP1847960A3 (en) * 2006-03-27 2008-01-23 General Electric Company Inspection apparatus for inspecting articles
CN103353726B (en) * 2006-03-27 2016-07-06 通用电气检查技术有限合伙人公司 For detecting the detecting device of article
US8310533B2 (en) 2006-03-27 2012-11-13 GE Sensing & Inspection Technologies, LP Inspection apparatus for inspecting articles
EP2145157A1 (en) * 2007-05-14 2010-01-20 Robert Bosch GmbH Method for displaying video images and video system
EP2185894A1 (en) * 2007-08-07 2010-05-19 Palm, Inc. Displaying image data and geographic element data
US9329052B2 (en) 2007-08-07 2016-05-03 Qualcomm Incorporated Displaying image data and geographic element data
EP2185894A4 (en) * 2007-08-07 2012-05-02 Hewlett Packard Development Co Displaying image data and geographic element data
US8994851B2 (en) 2007-08-07 2015-03-31 Qualcomm Incorporated Displaying image data and geographic element data
WO2009020785A1 (en) 2007-08-07 2009-02-12 Palm, Inc. Displaying image data and geographic element data
WO2009044343A2 (en) * 2007-10-05 2009-04-09 Nokia Corporation Method, apparatus and computer program product for multiple buffering for search application
WO2009044343A3 (en) * 2007-10-05 2009-05-28 Nokia Corp Method, apparatus and computer program product for multiple buffering for search application
CN101910936B (en) * 2007-12-24 2012-07-04 微软公司 Guided photography based on image capturing device rendered user recommendations
EP2225607A4 (en) * 2007-12-24 2010-12-22 Microsoft Corp Guided photography based on image capturing device rendered user recommendations
EP2225607A2 (en) * 2007-12-24 2010-09-08 Microsoft Corporation Guided photography based on image capturing device rendered user recommendations
US10942964B2 (en) 2009-02-02 2021-03-09 Hand Held Products, Inc. Apparatus and method of embedding meta-data in a captured image
US9959495B2 (en) 2009-06-12 2018-05-01 Hand Held Products, Inc. Portable data terminal
US9519814B2 (en) 2009-06-12 2016-12-13 Hand Held Products, Inc. Portable data terminal
US11042793B2 (en) 2009-06-12 2021-06-22 Hand Held Products, Inc. Portable data terminal
US8855929B2 (en) 2010-01-18 2014-10-07 Qualcomm Incorporated Using object to align and calibrate inertial navigation system
CN104221056A (en) * 2010-02-05 2014-12-17 英特尔公司 Method for providing information on object within view of terminal device, terminal device for same and computer-readable recording medium
EP2533214A4 (en) * 2010-02-05 2015-11-18 Intel Corp Method for providing information on object within view of terminal device, terminal device for same and computer-readable recording medium
US9488488B2 (en) * 2010-02-12 2016-11-08 Apple Inc. Augmented reality maps
US20170074675A1 (en) * 2010-02-12 2017-03-16 Apple Inc. Augmented reality maps
US11692842B2 (en) 2010-02-12 2023-07-04 Apple Inc. Augmented reality maps
US20110199479A1 (en) * 2010-02-12 2011-08-18 Apple Inc. Augmented reality maps
US10760922B2 (en) 2010-02-12 2020-09-01 Apple Inc. Augmented reality maps
EP2365283A3 (en) * 2010-03-10 2013-01-23 Astrium GmbH Information output device
EP2385500A3 (en) * 2010-05-06 2017-07-05 LG Electronics Inc. Mobile terminal capable of providing multiplayer game and operating method thereof
JP2013537616A (en) * 2010-06-10 2013-10-03 クアルコム,インコーポレイテッド Acquisition of navigation support information for mobile stations
WO2011156792A1 (en) * 2010-06-10 2011-12-15 Qualcomm Incorporated Acquisition of navigation assistance information for a mobile station
US9229089B2 (en) 2010-06-10 2016-01-05 Qualcomm Incorporated Acquisition of navigation assistance information for a mobile station
US9583074B2 (en) * 2011-07-20 2017-02-28 Google Inc. Optimization of label placements in street level images
US20150022555A1 (en) * 2011-07-20 2015-01-22 Google Inc. Optimization of Label Placements in Street Level Images
US10445772B1 (en) 2011-09-09 2019-10-15 Google Llc Label placement based on objects in photographic images
EP2610871A1 (en) * 2011-12-30 2013-07-03 Research In Motion Limited Methods and apparatus for use in providing object indicators during video recording or playback
US11170741B2 (en) 2012-02-29 2021-11-09 Nokia Technologies Oy Method and apparatus for rendering items in a user interface
US9208583B2 (en) 2013-02-13 2015-12-08 Blackberry Limited Device with enhanced augmented reality functionality
EP2767953A1 (en) * 2013-02-13 2014-08-20 BlackBerry Limited Device with enhanced augmented reality functionality
GB2615628A (en) * 2021-12-07 2023-08-16 Motorola Mobility Llc Electronic device that presents location-tethered augmented reality notifications

Similar Documents

Publication Publication Date Title
WO2005124594A1 (en) Automatic, real-time, superimposed labeling of points and objects of interest within a view
US10578452B2 (en) Method and apparatus for providing service using a sensor and image recognition in a portable terminal
US9721392B2 (en) Server, client terminal, system, and program for presenting landscapes
US8103126B2 (en) Information presentation apparatus, information presentation method, imaging apparatus, and computer program
CN107404598B (en) Imaging device, information acquisition system, program, and recording medium
US7088389B2 (en) System for displaying information in specific region
KR101423928B1 (en) Image reproducing apparatus which uses the image files comprised in the electronic map, image reproducing method for the same, and recording medium which records the program for carrying the same method.
US20140036129A1 (en) Photographing apparatus and photographing method
US20070228159A1 (en) Inquiry system, imaging device, inquiry device, information processing method, and program thereof
US20090278949A1 (en) Camera system and method for providing information on subjects displayed in a camera viewfinder
CN103685960A (en) Method and system for processing image with matched position information
JP2013507677A (en) Display method of virtual information in real environment image
US8339477B2 (en) Digital camera capable of detecting name of captured landmark and method thereof
JP2009267792A (en) Imaging apparatus
JPWO2007116500A1 (en) Information presentation system, information presentation terminal and server
JP2012038153A (en) Surrounding information retrieval device and surrounding information retrieval method
KR101397873B1 (en) Apparatus and method for providing contents matching related information
US9635234B2 (en) Server, client terminal, system, and program
US20110149089A1 (en) System and method for generating an image appended with landscape information
EP1730948A2 (en) Methods and apparatuses for formatting and displaying content
KR20110119179A (en) Method for updating panoramic image and location search service using the same
JP2008152374A (en) Image system, photographing direction specifying device, photographing direction specifying method and program
KR20160141087A (en) Providing system and method of moving picture contents for based on augmented reality location of multimedia broadcast scene
JP5969062B2 (en) Image processing apparatus and image processing method
JP5680358B2 (en) Imaging apparatus and imaging method

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KM KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NG NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Country of ref document: DE

122 Ep: pct application non-entry in european phase