WO1999040545A1

WO1999040545A1 - Sequential image storage system with pre-event history

Info

Publication number: WO1999040545A1
Application number: PCT/US1999/001810
Authority: WO
Inventors: Gary A. Rayner
Original assignee: I-Witness, Inc.
Priority date: 1998-02-09
Filing date: 1999-01-27
Publication date: 1999-08-12
Also published as: AU2478199A

Abstract

A system for recording sensor data generated during the time period surrounding a triggering event is disclosed. Such data can include video and audio signal data, as well as data from accelerometers, velocity meters, geographical coordinates, environmental conditions, vehicle control inputs, and the like.

Description

-1 - SEQUENTIAL IMAGE STORAGE SYSTEM WITH PRE-EVENT HISTORY

DESCRIPTION

Technical Field

The present invention relates to systems for storing sequential image and sound data and, more particularly, to such systems which store data from multiple sensors during the time period surrounding triggering events.

Background of the Invention

Various recorders have been used to gather and retain data obtained from sensors. One well known such recorder is the "black box" recorder typically carried by commercial and military aircraft. These devices record certain operating parameters of the aircraft, as well as the voices of the operators, and have found use in setting maintenance schedules, or to assist in determining the cause of an accident. Data recorders have been known to use magnetic tape or solid-state storage devices. However, such "black box" devices have not included the capability of storing video data, an important source of information for many events.

The ever-increasing use of motor vehicles has created a complex and expensive system of determining liability for the damages which result from accidents. Inevitably, it is difficult to reconstruct the events leading up to, and resulting from, the accident impact, as witness testimony is often unreliable and contradictory, and the physical remains generally do not contain sufficient evidence of the actual incident. A device to gather and record data from a wide range of operating parameters would be of great use, provided that the data covered the time leading up to the accident, as well as the accident itself, and that the data was secure so that it could be admitted in proceedings to determine liability.

In addition, tests such as staged accidents under controlled conditions are useful aids in designing safer vehicles, so long as they generate sufficient information to satisfy the design criteria. Due to its -2-

bulk and limited range of inputs, the "black box" type of recorder is not well suited to record data from small vehicles, anatomical test dummies and the like.

Gathering data that is generated by various sensors on smaller vehicles or objects has been done in the past by using either an umbilical cable or radio telemetry. Clearly, an umbilical cable imposes severe restrictions on mobility, and radio telemetry is subject to loss of data due to radio interference, misalignment of antennas or extremes in the test environment. In either event, these solution are impractical for reconstructing the events surrounding a routine traffic accident.

Recognizing the limitations of the "black box" devices in addressing these needs, efforts have been made to construct recording devices which are lighter, more versatile, and more secure in their data storage. For example, U.S. Patent No. 5,497,41 9 to Hill discloses a device for recording sensor data which included provisions to compress and uncompress data, as well as to encrypt the stored data to enhance the integrity of the data. U.S. Patent No. 5,056,056 to Gustin discloses a device to record various analog signal inputs occurring before and after a triggering event. U.S. Patent No. 5,581 ,464 to Woll et al. discloses an event recording apparatus to monitor the operational parameters of a vehicle. U.S. Patent No. 5,262,81 3 to Scharton discloses a device to trigger the operation of a camera mounted in a vehicle upon the occurrence of an impact. U.S. Patent No. 5,499, 1 82 to Ousborne also discloses an recording apparatus to monitor the operational parameters of a vehicle. U.S. Patent No. 5,31 1 , 1 97 to Sorden et al. discloses a device to trigger the operation of a location reporting device in a vehicle upon the occurrence of an event such as an impact. Each of these devices has certain limitations which reduce its applicability as a secure storage device for recording numerous sensor inputs occurring before and after a triggering event.

Thus, it is considered desirable to provide a compact recording device which is capable of recording a wide range of data inputs, including video data, from the time period surrounding a triggering event, such as an accident. It is also considered desirable to record the data in a reliable and secure manner, so that the veracity of the data would be unquestionable. DISCLOSURE OF THE INVENTION

The present invention provides a system for recording sensor data from the time period surrounding a triggering event. In one aspect, the system comprises a plurality of data sensors capable of generating a plurality of separate data streams in response to external inputs, wherein at least one of said sensors responds to optical inputs by generating a data stream comprising digital images, a memory means for recording said plurality of data streams generated by said data sensors, a continuous loop storage means for temporarily retaining said plurality of data streams, a control circuit means for controlling the input of said plurality of data streams into said memory means, and means responsive to a triggering event for terminating the input of said plurality of data streams after a predetermined period of time. Further aspects of the invention provide such systems capable of responding to audio inputs and motion inputs, as well as systems which include means for displaying preselected portions of the digital image data stream.

A selected embodiment of the present system employs a two-tier system which comprises a first data storage means operating as a recirculating buffer and a second data storage means for receiving the contents of the first data storage means after the recording into the first data storage means has been terminated in response to the triggering event.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the present invention will now be described with reference to the attached figures wherein: Figure 1 depicts a block diagram of an embodiment of the functional elements of a system in accordance with the present invention;

Figure 2 depicts a block diagram of a preferred embodiment of a system in accordance with the present invention;

Figure 3 depicts a block diagram of an algorithm used to perform the continuous loop recording mode of the present invention; -4-

Figure 4 depicts a block diagram of an algorithm used to perform the pause and user control mode of the present invention; and

Figure 5 depicts a block diagram of an algorithm used to perform the video playback mode of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a system for recording sensor data from the time period surrounding a triggering event. In one aspect, the system comprises a plurality of data sensors capable of generating a plurality of separate data streams in response to external inputs, wherein at least one of said sensors responds to optical inputs by generating a data stream comprising digital images, a memory means for recording said plurality of data streams generated by said data sensors, a continuous loop storage means for temporarily retaining said plurality of data streams, a control circuit means for controlling the input of said plurality of data streams into said memory means, and means responsive to a triggering event for terminating the input of said plurality of data streams after a predetermined period of time. As an overview, the present system is designed to constantly capture and record a sequence of still picture images from a camera into an infinite loop digital memory. When the system is triggered by a trigger input signal, typically an accelerometer or "G" force sensor, from a triggering event, typically an impact, the present system will save a predetermined number of picture frames from the period prior to the triggering event, and continue recording additional frames until a predetermined time has elapsed or a predetermined number of frames have been captured. Once the predetermined number of frames before and after the trigger event have been captured, the memory buffer is automatically copied to a permanent backup memory facility to prevent loss or erasure due to loss of power. It is also considered desirable that this permanent backup memory facility be secure, so that the recorded data can not be altered.

One useful feature of the present system is that it is intended to capture, record and provide a playback of events which occurred before and after a vehicle accident or impact, and that the image recording memory is implemented as a continuous loop until the trigger event occurs. The trigger event may be caused by an impact sensor, or panic button. A key feature of this system is the ability to record and preserve a sequence of still picture images prior to the triggering event. The present system may also provide a video output to enable viewing of the sequence of still frames, or display a step by step viewing of the captured sequence.

As disclosed above, a selected embodiment of the present system employs a two-tier system which comprises a first data storage means operating as a recirculating buffer and a second data storage means for receiving the contents of the first data storage means after the recording into the first data storage means has been terminated in response to the triggering event.

In this embodiment, the system initially enters the continuous loop still image recording mode to capture images into an endless loop video memory. Frames are recorded sequentially at a predetermined interval (Figure 3).

At the start of the loop, a still picture frame is captured into memory. An interval time counter will determine when to advance the frame pointer. If the time has elapsed to advance the frame, the frame pointer will be incremented to the next frame position. The memory has a finite limit on the number of frames that may be stored in it. When the capacity of the memory is reached as it is filled with images, the memory frame pointer will be reset back to the start of the memory, so that the capturing of the sequence of still images may continue.

The loop will check to see if the event trigger has been tripped. If it has, a check must be made to ensure that the desired number of frames before the trigger event are preserved and not written over by new, post- trigger frames. This is accomplished by only allowing a certain number of frames to be recorded after the trigger event, therefore preventing overwrite of the preserved pre-trigger frames.

When the desired number of frames has been captured after the trigger event, the memory is copied to a permanent memory to prevent accidental erasure. The system will then go into a pause mode, waiting for the user to retrieve or display the captured video images. -6-

After the trigger event has caused images to be captured into memory, the system waits for user control. Control instruction may be from push buttons, or a digital control signal (Figure 4) . Typical actions that may be performed by the user include: Displaying a still frame, advancing the still frame to the next captured frame, reversing the still frame to the previous captured frame, playing a sequence of still frames at an interval rate, and erasing the memory and re-entering the continuous loop record mode. Each of these functions can be implemented as follows: The system will display the still image at the current frame pointer.

A frame advance instruction will cause the frame pointer to be incremented, therefore displaying the next recorded image frame. Boundary checking is implemented to ensure correct memory address wrap around, and to prevent the user from advancing past the last captured frame.

A frame reverse instruction will cause the frame pointer to be decremented, therefore displaying the previous captured image frame. Boundary checking is implemented to ensure correct memory address wrap around, and to prevent the user from reversing past the first captured frame.

A play sequence instruction will cause the system to enter the play mode and play a sequence of the captured still frames.

An erase instruction causes the video memory to be erased and initialized, and then sets the system into the continuous loop record mode. Further aspects of the invention provide such systems which include means for displaying preselected portions of the digital image data stream. This mode plays a motion sequence of the recorded still image frames. Frames are played back sequentially at a predetermined interval (Figure 5). The system will display the still image at the current frame pointer.

An interval time counter will determine when to advance the image frame pointer. If the time has elapsed to advance the frame, the frame pointer will be incremented to the next frame position.

The memory has a finite limit on the number of still image frames that can be recorded. When the capacity of the memory is reached as it is displaying the frame, the memory frame pointer will be reset back to -7-

the start of the memory again so that the display of the consecutive still images may continue.

When a user instruction is received, or when the last captured frame has been displayed, the system will go back to the pause/user control mode to wait for further instructions.

In practice, the present system will be constructed of various commercially available components, and will optionally include a case for housing the various components of the system. Data will be introduced into the system typically through a multi-pin connector mounted on the case, while processed and recorded data is available through that same multi-pin connector, or another output connector.

In a selected embodiment of the invention, the present system can be constructed in reference to the drawings, where Figure 1 shows a block diagram of a system 1 0 in accordance with the present invention, with greater detail provided in Figure 2.

In most cases, the input data to the system will be in digital form, either produced from the sensor, or more generally by resort to a analog- to-digital (A/D) converter. To accommodate a wide range of sensors with unknown offsets and gains, analog signals means can be provided for modulating the input signals prior to processing by the A/D converter. A circuit useful in this regard will include a first amplifier having a gain adjust potentiometer, a second amplifier having an offset adjustment potentiometer, and an anti-aliasing filter circuit. In the preferred embodiment, however, the input ranges and offsets of the sensors will be known. In this circumstance, an anti-aliasing filter circuit will prove useful. The output of each such circuit comprising the modulated signal provides an analog signal whose voltage output has been modified to bring it within the limits of the A/D converter circuit.

A start-conversion control circuit and an end-conversion control circuit will serve to synchronize the operation of the A/D converter and the memory. The analog input data must be converted to digital form to generate a data stream before that data is applied to and recorded in the buffer memory, and these circuits insure that the conversion sequence has been completed before the memory records the digital data stream. In the preferred embodiment, however, the microprocessor of the present -8-

invention will direct and control the data acquisition functions, and the image sampling function will be performed utilizing digital control circuitry.

The present system will also include a digital-to-analog converter circuit to convert certain data recorded in the system, for example video data, into a video image which can be displayed on a video monitor or television.

The A/D converter converts each analog signal to a digital signal which appears as a data stream on a data bus. These data streams are applied to the memory device which serves as the continuous loop buffer. Typically, such a memory device can be a relatively inexpensive fast DRAM or SRAM, preferably 1 6 megabit, which is generally available commercially such as, for example, as part number UPD4251 7400L-A80 from NEC Corporation.

In the event of a triggering event, data from the buffer memory will be copied to the second tier of the memory system. This second tier will be non-volatile memory, such as EEPROM, NOR memory, Flash ROM, or a battery-backed static RAM. Desirably, in order to increase the reliability of the system, the non-volatile memory will be FLASH memory, which does not require a battery to maintain its integrity. Typically, such a FLASH memory device will be 64 megabit, which is generally available commercially such as, for example, as part number G28F640J5-1 50 from Intel Corporation. Of course, other means of storage will prove useful in certain applications, such as hard disk memory, floppy disk memory, digital tape and the like. The memory device is controlled by a memory controller circuit

(shown in greater detail in Figure 2). Typically, these and other control functions will be implements by a central processing unit (CPU) containing the software which controls all operations of the system. In the present invention, the continuous loop memory is configured to record 8 bit data signals in separate channels and to permit access to that data on the output circuit without destroying the data itself. If necessary for certain types of memory, battery backup system can be provided to insure that the data in the memory is retained, even when the data recorder is not connected to an external source of power. In the preferred embodiment, however, the battery backup system will not be necessary to retain the data stored in memory, rather it can prove useful to allow the completion of the recording of the triggering event and its aftermath in the event that the main power supply is damaged or otherwise rendered unavailable. As an additional function, the battery backup system will provide sufficient power to permit transfer of the data from the first tier to the second tier of the data storage system, e.g. from the DRAM to the Flash memory. In addition, it is contemplated that improvements in memory devices will likely be available to replace the DRAM-type device, which will not suffer memory loss with the loss of power. Under such circumstances, a battery backup system will generally be unnecessary, as perhaps the second tier of memory as well.

A memory controller circuit provides the means for controlling the flow of data into the memory means whereby new data is sent to said memory means and old data is overwritten. As shown in Figure 2, this circuit includes an address controller which incoming data is to be recorded. As new data is read into the memory, the old data at that memory location is overwritten. When the highest memory address is reached, the address controller will direct the next byte of data to the first or lowest memory address. Thus, the buffer memory is made to act like a recirculating memory device or a continuous loop recording. The memory controller circuit also includes several circuits that control the operation of other systems. The main clock provides the timing pulses necessary to the transfer of data into and out of the buffer memory. The output of the clock may be divided by the sample rate selector into one of a number of sample rates, selectable by the user or preset and secured.

In a preferred embodiment, multiple data acquisition rates are accommodated by sampling and recording the data streams at different rates. For example, the data stream from an audio sensor can be sampled at 1 5.625 KHz or 1 5,625 samples per second. An accelerometer sensor for measuring G-force data can be sampled at 1 00 to 700 samples per second, as determined by the software. The actual recording rate for such data could simply record the maximum G-force experienced, e.g. four times per second. The image data stream will generally be sampled at the slowest rate, typically ranging from 1 to 60 samples per second as desired, typically four times per second will prove sufficient. -1 0-

The data acquisition rate will also affect the amount of time data can be recorded before the previously-entered data is overwritten. At the rates disclosed above, the device of the present invention will typically store 80 frames of image, together with the associated audio and G-force data, for a total of 20 seconds of event history. However, it is again anticipated that improvements in high-density, low-cost memory will enable substantial increases in capacity at minimal additional cost. Each data stream will be continuously written into the buffer memory until a signal is generated in response to a triggering event. As a result, only a predetermined amount of data (depending on the sampling rate) is actually retained since the prior data is constantly being overwritten. Accordingly, circuit means in the form of a sequence controller is provided. The controller is responsive to the occurrence of the triggering event and provides a signal to the control circuit means or memory controller circuit to limit the input of new data into the memory means to a predetermined time after the triggering event, or a predetermined number of additional samples.

The sequence controller will include a circuit which responds whenever a start signal has been generated to initiate the operation of a record trigger to act as a counter. The counter could also be controlled by the main clock, and when an output from the counter is generated, all further writing of data into the memory is terminated and the data that is in the memory is automatically copied to the second tier memory. The start signal can be user generated. It might be a signal uniquely generated by the user, or it might be derived from one of the input signals. For example, one of the input signals might represent acceleration or "G" force, and the triggering event might be defined as whenever the acceleration exceeds some predetermined value.

Once the record trigger has generated an output, no more data may be written into the memory. An output from the record trigger is applied to the read/write controller to inhibit further writing of data.

Power may be provided to the present system either externally from the power supply of the vehicle in which the system is installed, optionally to charge an internal battery, or by an internal battery, such as a NiCad of Lithium ion battery. If external power is being used, a regulator charger could be used to control the charging rate for the optional battery to -1 1 -

maintain it at maximum standby capacity. If the external power is eliminated, then power to continue operation of the system could be provided by the optional battery through such a regulator.

To read the data stored in the memory, a signal is sent to the read counter. This device causes the read/write controller to interrogate the memory, and the data therein is written to a data bus and through the data bus buffer to the output connector. The recorded data may be read as many times as desired without affecting the memory since the reading operation is non-destructive. In the preferred embodiment, the recorded data streams are output in "correlated packets" of data that are associated with the sample time for a single image frame. For example, if the memory playback is advanced by a single frame, then in the four-frames-per-second example 0.25 seconds of audio data and the related G-force data will also be retrieved. For the other rates disclosed above, different sized packets will be obtained and played. When the device retrieves data at a "normal" playback mode, the data display will present a substantially continuous, substantially "real time" stream.

The read counter outputs an equivalent number of pulses as there are addresses in the memory. In this manner, each time the read counter is activated, the address counter will read all the data in the memory, starting at its present location, reading to the highest memory address, and then reading from the lowest memory address back to the present memory address. However, there may not be a precise one-to-one correspondence between the stored data and the memory addresses. The address where data recording was terminated will be determined by the occurrence of the start signal and the setting of the record trigger.

A reset circuit will be employed to reset the start circuit and permit dam again to be recorded in the memory. Since it is important that the data in the memory be protected against inadvertent loss, the reset circuit will require a unique input signal before it outputs a reset instruction.

The data sensors of the present system will generally include sensors for video and audio signal data, as well as data from accelerometers, velocity meters, geographical coordinates, environmental conditions, vehicle control inputs, and the like. The video sensor will typically be one or more image cameras, such as a CCD or Silicon sensor -1 2-

electronic camera. The audio sensor will typically involve a microphone positioned to receive audio inputs from either interior or exterior sources. Time sequence information can be obtained from the master clock of the system and be stored and accessed in sequence with the retrieved data. Other sensors will be provided, for example, through accelerometers for determining "G" force in any of the three dimensions, and the electronic management systems common in contemporary vehicles can be employed to provide data streams to monitor vehicle control inputs such as the accelerator position, brake inputs, turn signals, steering wheel position and the like. Geographical positioning information can be obtained for example by reference to the GPS satellite system.

Certain additional features can easily be provided within the framework of the present system. For example, the data streams can be compressed to increase the effective storage capacity of the memory systems. The audio data can be synchronized with the video data, and, in fact, any data stream can be synchronized with any other stream to enhance its usefulness.

All patents and patent applications cited in this specification are hereby incorporated by reference as if they had been specifically and individually indicated to be incorporated by reference.

Although the foregoing invention has been described in some detail by way of illustration and example for purposes of clarity and understanding, it will be apparent to those of ordinary skill in the art in light of the disclosure that certain changes and modifications may be made thereto without departing from the spirit or scope of the appended claims.

CLAIM

Claims

-1 3-CLAIMS

1 . A system for recording sensor data from the time period surrounding a triggering event comprising: a plurality of data sensors capable of generating a plurality of separate data streams in response to external inputs, wherein at least one of said sensors responds to optical inputs by generating a data stream comprising digital images; memory means for recording said plurality of data streams generated by said data sensors; continuous loop storage means for temporarily retaining said plurality of data streams; control circuit means for controlling the input of said plurality of data streams into said memory means; and means responsive to a triggering event for terminating the input of said plurality of data streams after a predetermined period of time.

2. A system as recited in claim 1 wherein at least one of said plurality of data sensors responds to audio inputs by generating a data stream comprising digital audio data.

3. A system as recited in claim 1 wherein at least one of said plurality of data sensors responds to motion inputs by generating a data stream comprising acceleration data.

4. A system as recited in claim 1 further comprising means displaying preselected portions of the digital image data stream.

5. A system as recited in claim 1 wherein said continuous loop storage means employs a two-tier system which comprises: first data storage means operating as a recirculating buffer for continuously recording the plurality of data streams until a predetermined limit is reached before recording over the earliest portion of the data stream; and -1 4- second data storage means for receiving the contents of said first data storage means after the recording into said first data storage means has been terminated in response to the triggering event.

6. A system as recited in claim 5 wherein said first data storage means comprises a first digital memory means.

7. A system as recited in claim 6 wherein said first digital memory means comprises static RAM or fast DRAM memory.

8. A system as recited in claim 5 wherein said second data storage means comprises a second digital memory means.

9. A system as recited in claim 8 wherein said second digital memory means is permanent, non-volatile memory.

1 0. A system as recited in claim 9 wherein said permanent, nonvolatile memory is selected from the group consisting of FLASH memory, battery-backed static memory, NOR memory, EEPROM memory, hard disk memory and floppy disk memory.

1 1 . A system as recited in claim 1 wherein the predetermined period of time after the triggering event is selectable.

1 2. A system as recited in claim 1 wherein said means responsive to a triggering event comprises means for sensing a change in the data stream generated by at least one of said data sensors.