WO2008065257A1

WO2008065257A1 - Techniques for improving reliability of a prediction system

Info

Publication number: WO2008065257A1
Application number: PCT/FI2007/050651
Authority: WO
Inventors: Kari Vasko; Pauli Misikangas
Original assignee: Ekahau Oy
Priority date: 2006-11-30
Filing date: 2007-11-30
Publication date: 2008-06-05
Also published as: FI20065766A0

Abstract

A context estimation system observes physical quantities at a target object site (21 ) and outputs a context estimate which is used to trigger context-related actions. Each context (22 - 24) defines circumstances for the observations of the physical quantities. Calibration data (25), made under these circumstances, is used to build context models. The context models are used by a score function which indicates a likelihood that a given observation is made under a given context. Similarities between two context models are identified by selecting two contexts, selecting an observation from the calibration data of the first context; calculating a first score with the selected observation and the first context; calculating a second score with the selected observation and the second context; and updating the similarity using the first score and the second score. The similarity can be used to make changes affecting the performance of the context estimation system.

Description

Techniques for improving reliability of a prediction system

Information on related application

[0001] This application claims priority from Finnish patent application 20065766 the contents of which are incorporated herein by reference.

Background of the invention

[0002] The invention relates generally to techniques for improving predictability of unknown values, particularly when such techniques are used in control of a physical system. The invention relates particularly to a context estimation system for an environment including at least one observable context-dependent physical quantity. The context estimation system is capable of outputting a context estimate for a target object using an observation made by a sensing device attached to the target object, wherein the observation comprises values for one or more physical quantities, and wherein the context estimate is used by a context-aware application to trigger context-related actions. [0003] A method for implementing such a context estimation system comprises defining two or more contexts, wherein each context defines the circumstances under which observations of the physical quantities are made. The method further comprises providing calibration data for each context, wherein the calibration data for a context comprises several observations, each of which is made under the circumstances defined by the context and comprises values for one or more physical quantities. Then a context model is constructed for each context using the calibration data of the context, wherein the context models are used by a context score function taking an observation and a context as input and returning a score indicating the likelihood that the given observation is made under the given context.

[0004] In order to facilitate understanding the invention, two illustrative but non-restricting application examples of such a physical systems will be presented. In one application example the context relates to the location of a target object in the environment. Location estimation is carried out by a position- ing system, which is a non-restricting example of a context estimation system. The positioning system is taught to estimate a target devices' location given a set of values of location-dependent physical quantities observed at the target device's location. Such techniques are described in commonly owned patent applications, some of which are listed at the end of this description. Another illustrative but non-restrictive practical application of the invention relates to automatic plant growing in a greenhouse. A generic problem related to such techniques is that the model-building step may not provide an adequate estimate of the reliability of the context estimation capabilities of the context estimation system. [0005] In general, prediction of unknown values of interest is, ultimately, a purpose of all statistical data analysis. In order to get a reliable understanding of a predictive performance of a prediction system, it is essential to asses various aspects of the system. Traditionally, predictive performance of the prediction system is estimated on the basis of test data. Those skilled in the art know these techniques as train-and-test, bootstrap and k-fold cross-validation techniques.

[0006] It is, however, possible to assess predictive aspects of the system using calibration data only without leaving out a part of the training data as test data, as follows. Calibration data is collected under different conditions, or more generally contexts, say C[1], C[2], ..., C[k], and for each context C[i] a context model M[i] (i=1 ,2,...,k) is taught given a calibration data D[i] of the context C[i]. Calibration data D[i] is used to teach a context model M[i] and the whole prediction systems consists of the context models M[1], M[2]...,M[k]. After that, it is assessed how similar the context models M[1], M[2] M[k] are. If there is a subset of context models that are similar, it is an indication that the context estimation system is not able to successfully discriminate the corresponding context and, accordingly, there is an increased risk for false predictions. Known techniques for assessing performance of a prediction system in such a setting using calibration data only include ad hoc distance-based methods and Kullback-Leibler divergence (KL-divergence) technique and its derivatives. Ad hoc distance-based methods are based on evaluating a distance operation either on the raw signal data space or on the derived parameter space. For instance, an implementation of a such a method could evaluate an averaged distance of the model parameters between the different context or difference between the corresponding probability distributions. KL-divergence and its derivatives are theoretically more solid and they are based on evaluating a distance measure between the probability distributions. In this application an object would be to evaluate distance between probability distributions of different contexts. Both of the specified prior-art approaches suffer from certain limita- tions, however. Brief description of the invention

[0007] An object of the invention to alleviate one or more of the problems identified above. This object is achieved by develop a method, an apparatus and software products as defined in the attached independent claims. Specific em- bodiments of the invention are presented in the dependent claims and in the following description.

[0008] An aspect of the invention is a method for generating a computer- implemented context estimation system for an environment including at least one observable context-dependent physical quantity, wherein the context esti- mation system comprises a data processor and memory and is capable of out- putting a context estimate for a target object using an observation made by a sensing device attached to the target object, wherein the observation comprises at least one value for one or more physical quantities, and wherein the context estimate i used by a context-aware application to trigger one or more context-related actions, the method comprising: a) defining two or more contexts, wherein each context defines the circumstances under which observations of the physical quantities are made; b) providing calibration data for each context, wherein the calibration data for a context comprises several observations, each of which is made under the circumstances defined by the context and comprises values for one or more physical quantities; c) constructing a context model for each context using the calibration data of the context, wherein the context models are used by a context score function taking an observation and a context as input and returning a score which indicates a likelihood that the inputted observation is made under the inputted context; d) generating a similarity report identifying similarities between context models, wherein said generating the similarity report comprises:

• selecting a first context and a second context; • selecting an observation from the calibration data of the first context;

• calculating a first score using the context score function giving the selected observation and the first context as input;

• calculating a second score using the context score function giving the selected observation and the second context as input; and • updating the similarity report using the first score and the second score; e) making real and/or simulated changes affecting the performance of the context estimation system by utilizing the similarity report; and f) storing the context models and the context score function in the memory of the context estimation system, whereby the context estimation system is capable of outputting a context estimate for the target object, wherein the context estimate is determined using the scores returned by the context score function when given an observation made by a sensing device attached to the target object as input.

[0009] Another aspect of the invention is a set of computer-readable media for a data processing system which comprises a model construction section and a context estimation section. The set of computer-readable media comprises first computer program instructions whose execution in the model construction sec- tion causes the model construction section to carry out steps a) through d) of claim 1 ; and second computer program instructions whose execution in the context estimation section causes the context estimation section to carry out step f) according to claim 1. [0010] Yet another aspect of the invention is a data processing system which comprises a model construction section and a context estimation section, wherein the data processing system further comprises the above-defined set of computer-readable media.

[0011] Some implementations of the present invention aim at providing improved analysis of similarity of competing statistical models. The focus is not in model selection but introduction of a new approach to assess ambiguity of the statistical prediction system. The inventive approach is based on a new similarity measure called "T-similarity". Various specific embodiments and implementations of the invention relate to local and global measures of ambiguity of the prediction system that are based on the T-similarity measure. Other embodi- ments and implementations relate to a generalization of the T-similarity measure called (T,f,s)-similarity measure that can be used to assess how noise tolerant the prediction system is.

[0012] For a location-estimation application, the invention can be implemented as a computer-implemented method for providing a reliability estimate relating to a property set of a target device in an environment, wherein the property set comprises location. The method comprises, prior to determining the property set of the target device, determining a value for each of one or more location- dependent physical quantities in several locations in the environment, wherein the values for the one or more location-dependent physical quantities are determined by calibration measurements and/or simulation. The method further comprises using the determined values for the one or more location-dependent physical quantities to teach a statistical model to model the environment such that the statistical model is operable to estimate the property set of the target device given a set of location-dependent physical quantities observed at the target device's location. The statistical model so taught is used to provide a reliability estimate relating to the property set of the target device in the environment.

[0013] As a concrete but non-restricting example, consider a positioning task where a positioning system models a positioning environment which comprises two regions called A and B. In this example the positioning system is an exam- pie of a context estimation system and a region is an example of a context. Those skilled in the art will realize that the invention can be generalized to an arbitrary number of regions. A context may also be defined as a composition of other contexts, for instance a set of regions. The aim or the positioning system is to predict whether the target object is inside the region A or B. A first phase in the setup of the positioning system comprises collecting calibration and/or simulation data from regions A and B. A second phase comprises teaching a positioning model coupled to the positioning system on the basis of the calibration data. A third phase comprises collecting test data in order to get a rough understanding of the predictive performance of the system. A problem in this scenario is that the positioning model coupled to the positioning system must be more or less completed (taught on the basis of the calibration or simulation data) before its performance can be assessed. If it turns out that the positioning model comprises ambiguous regions, the model setup must be started with new parameters. Thus, it would be beneficial to know whether the positioning model which was taught on the basis of the calibration data contains ambiguous regions (regions similar to one another), before the test data is collected. This is because in the real-world cases it is not realistic to assume that extensive test data will always be available from all regions (or more generally from all contexts). For instance, if the model comprises 1000 regions, a huge amount of resources will be needed to collect reliable test data from every region. Further, in many practical positioning applications it is useful to know how noise-tolerant the system is prior to applying any test data to it. Information regarding noise tolerance is essential, if it is likely that the signal values will be over-dispersed with respect to the calibration conditions. For instance, in hospitals, it is not possible to collect calibration data from the operation rooms while they are being used for medical operations, and WLAN signal strength data values are over-dispersed during actual use of the WLAN network. [0014] The inventive technique can be used in an automatic detection of problematic contexts, eg, regions in which the context estimation system, such as a positioning system, is likely to be unreliable. The inventive technique can use actual test data but is does not necessarily need any. In other words, the automatic detection of problematic regions can be performed solely on the basis of calibration or simulation data. In addition, the invention gives a technique to assess model-based noise-tolerance of the system via a (T,f,s)-similarity measure. This is valuable in order to make sure that the context estimation system is reliable and has a high noise tolerance level.

[0015] In one non-restrictive implementation example, the inventive technique can automatically detect contexts, such as regions, where an increased risk for large errors exists on the basis of calibration data and the context models. The inventive technique is more generic, as will be seen later in this document. A positioning application will be used as an illustrative but non-restrictive practical application.

[0016] Let us suppose, without a lack of generality, that the environment to be modelled consists of two contexts A and B. For example, let context A be "zone A", and context B is "zone B". The notion of context may be more gen- eral than a spatial location or region, e.g. an environmental condition or time interval. In other words, spatial location is covered by the term "context" used in the following description. Let us denote by D(X) calibration measurements that were recorded in the context X, where X is either A or B. Let ContSys be a context estimation system of the invention, wherein the positioning system is a non-restricting example of a context estimation system.

[0017] ContSys gets as an input a measurement of the explaining variables, eg WLAN signal strengths, and its output is a predicted context. The context estimation system chooses the context that has the maximal score and, thus, it holds that ContSys(x) = A if and only if score(A.x) > score(B,x), [1] [0018] where x is a measurement and score(Y.x) denotes a score of a context Y given the measurement x. Those skilled in the art know that embodiments of such score functions include an expected utility, conditional probability, fuzzy logic approaches and rough set theory. From now on, we will use as an illustrative, but not restricting example, the statistical decision theory. Those skilled in the art know that in the statistical decision theory the optimal decision is to choose a prediction that has maximal utility. We assume, for simplicity's sake, the classic 0/1 -utility function, which yields

ContSys(x) = A if and only if Pr(A|x) > Pr(B|x), [2] where x is a measurement and Pr[X[Y] denotes a conditional probability of an event X given the information Y. [0019] Recall that the Bayes theorem yields:

Pr(A|x) - Pr(A,x)/Pr(x) = Pr(A)Pr(x|A)/(Pr(A)Pr(x|A) + Pr(B)Pr(x|B)). [3]

[0020] The interesting quantities in the above equations (2) and (3) are the conditional marginal likelihoods Pr(x|A) and Pr(x|B). They both involve implicitly summing over all possible parameter values of the parameter space of the context A/B. In practice, this summation is rarely possible and, thus, in many implementations the maximum likelihood parameter values are fixed for each context. This issue will be described in more detail later in this document. [0021] Next, an enhanced implementation of the inventive idea will be described. In short, this implementation is based on answering to the following question 1 : "If the calibration data of the context A was given to the context estimation system, how often would ContSys predict the wrong context B?" [0022] This implementation is able to exploit training data as semi test data, since calibration data of the context A is not used in the construction process of the context B's model. On the other hand, the model of the context A is based on its own calibration data and, accordingly, it is expected that ContSys should give a higher score for the context A than for the context B. If this does not hold, contexts A and B are mixed up with one another. [0023] The inventive prediction technique can answer the above-presented question 1 by calculating the following probability quantity:

[0024] Pr[x is classified to context B | x is a calibration sample of A]= Pr[Pr(A|x) < Pr(B|x)|x is a calibration sample of a A] = similarity(A→B|D(A)). [4] [0025] Herein, x is a random observation from A's calibration data set D(A). It will be apparent that the higher the probability quantity is in equation [4], the more probable it is in the light of the calibration data that the context A is mixed up with the context B. An exemplary technique to compute a similarity measure defined in equation [4] is simply to loop over all observations of the context A and then calculate a proportion how often ContSys made a wrong prediction context B. [0026] The following quantity is known as the (Bayes) classification error:

Pr[ContSys makes prediction error] = Pr[x is classified to context B | x was collected in the context A] +

Pr[x is classified to context A | x was collected in the context B] =error(A,B),

where x is test data. Observe that if x calibration data it holds that

similarity(A->B|D(A)) + similarity(B→A|D(B))=error(A,B) under a closed-world assumption according to which the world consists of the contexts A and B.

[0027] Another illustrative but non-restrictive application of the invention relates to automatic plant growing in a greenhouse. In this example a target object is a plant and an environment is a greenhouse. For each plant species there is a set of contexts that describe the current status of the piant species. Illustrative but non-restrictive examples of the contexts of a plant species are A='dry\ B='suitable' and C='nutrient substance is needed'. Illustrative but non- restrictive examples of the corresponding context-dependent physical quantities are local ground moisture and a colour of a plant. For each plant, moisture and colour is measured independently in order to estimate a context of a plant by the plant context estimation system. In this example, context A refers to a status such that a plant requires more irrigation, context B indicates that the plant requires no immediate irrigation and, finally, context C indicates that the plant needs extra nutrient substances. For instance, if the plant context estimation system outputs for a specific plant that its context is simultaneously in the contexts A and C then context-aware application should trigger an irrigation process such that water and extra nutrient substances are delivered for the plant automatically. Calibration data in this example consists of the calibration measurements of the ground moisture and colour of a plant given the correct context. It is important that the plant context estimation system can make a clear difference between contexts A, B and C in order to maintain successful automatic plant growing greenhouse. In this application collection of the test data is a laborious process and, thus, it is a great advantage if the context estimation process can be improved without collecting actual test data. [0028] Next, a new concept called T-similarity, which is based on the previously defined similarity measure, will be defined. We will explain how the T-similarity can be utilized in identifying model-based similarities and noise tolerance. Techniques based on T-similarity are applicable to a wide variety of data analysis problems which involve a prediction task. The definition of T- similarity is as follows. Let 0<T<1 be a fixed real number. A context A is T- similar with a context B if a following inequality is true:

similarity(A → B|D(A)) > T. [5]

[0029] Note that similarity(A→A|D(A)) = 1.0 always and, accordingly, a context is always T-similar with itself for all 0<T<1. Note that T-similarity is not a sym- metric relation unlike, for example, the classification error. This means that if context A is T-similar with context B, it does not follow that context B is T- similar with context A. In this sense the measure is similar to the Kullback- Leibler divergence (KL-divergence). [0030] Methods based on T-similarity overcome some of the limitations of the existing methods. For instance, the above-mentioned KL divergence, unlike T- similarity, does not take into account the extra information that is available via calibration data set swaps. Second, unlike T-similarity, ad hoc-distance based methods are limited to some specific notion similarity induced by the distance metric, whereas the inventive similarity-based technique is model-based and, accordingly, is not limited to linearity or the like. Third, unlike commonly known train-and-test methods, the methods based on T-similarity do not need any genuine test data which is not used to teach a global prediction model. Thus, all calibration data can be used to construct the model prior to its validation. Fourth, if the data-generating mechanism is autocorrelating one, known split- based train-and-test techniques are not even applicable to assess realistic predictive performance of the prediction system, unlike methods based on the inventive T-simifarity measure. Yet further, the proposed T-similarity measure can utilize both the calibration data and the test data. Given that only calibration data is available, the inventive technique can still identify, for instance, zones that are potentially similar. Brief description of the drawings

[0031] Specific embodiments, implementations and practical applications of the invention and illustrative examples will be described in connection with the attached drawings, in which Figure 1 illustrates a general operational scheme in which the invention is used under three different contexts;

Figure 2 illustrates an example in which the context is determined on the basis of signal strengths in a WLAN network;

Figure 3 illustrates how the measure similarity(A→B|D(A)) may be constructed; Figure 4 shows a table which illustrates processing of signals disturbed by noise;

Figure 5 shows an exemplary section of a WLAN network with two access points and two contexts;

Figure 6 illustrates calibration data collected in the two contexts shown in Fig- ure 1 versus signal strength of the two base stations;

Figure 7 is a variation of Figure 6, wherein the observed signal strengths are manipulated in order to simulate systematic blocking of the access point signals;

Figure 8 is a redesigned version of the network section shown in Figure 5; and Figure 9 is a diagram which shows that the redesigned version of the network section shown in Figure 8 is robust with respect to systematic blocking of the access point signals.

Detailed description of specific embodiments

[0032] Figure 1 illustrates a general operational scheme in which the invention is used under three different contexts. The invention is not limited to any particular number of contexts, however. C(x) , wherein x is A₁ B or C, denotes context x, D(x) denotes calibration data collected under the context C(x), while M(x) denotes the context model that is taught using the calibration data D(x). It is not necessary to use calibration data only. For instance, it is possible to im- plement the invention in such a manner that D(x) in Figure 1 represents simulated values of the explanatory quantities of interest. In another implementation of the invention, D(x) may contain calibrated and simulated values of the explanatory values of interest which are combined statistically. Illustrative but non-restrictive examples of the explanatory quantities of interest include signal strengths in WLAN network, relative humidity of air, colour of a plant and opti- cal signals under different conditions. The contexts C(x), calibration data D(x) collected under the context C(x) the models M(x) taught using the calibration data D(x) are collectively denoted by reference numeral 11. Reference numeral 12 denotes a data set which results from an evaluation of the swap scores. Reference numeral 13 denotes a set of calculated T-similarrties.

[0033] Figure 2 illustrates an example in which the context is determined based on observed signal parameter values, such as signal strengths. A WLAN network comprises one or more access points, one of which is denoted by reference numeral 20. A target object 21, shown illustratively as palmtop computer, observes signals parameter values, such as signal strengths in the WLAN network, including signals from the access point 20. This example involves determining one of three contexts A, B and C, denoted by respective reference numerals 22, 23 and 24, on the basis of the observed signal strengths. [0034] Locating a receiver in a wireless communication environment per se is extensively discussed in commonly owned patent applications, some of which are listed at the end of this specification. In this positioning example, the location of the target object, or the signal strength sensing device attached to it, is an example of a context, while the signal parameter value for a specific radio transmitter, such as signal strength, is an example of the context-dependent physical quantity. A non-restrictive example of a context-aware application in this scenario could be an automatic lighting controlling system which controls the lighting in regions A, B, and C according to the context estimate of the target object, wherein lights are turned on only for the region indicated by the context estimate.

[0035] Reference numeral 25 denotes three sets of calibration data for the three contexts A, B and C. The calibration data sets are as follows D(A)={-60, -59, -65}; D(B)={-58, -57, -62}; and D(C)={-80, -85, -82, -81 , -60}.

[0036] Let us also suppose, for the purposes of this example, that the signal strength values are modellable by a normal distribution such that the expectation can be derived from the calibration data and the deviation is constant "5" over all contexts. Recall that the probability density of a value x given the nor- mal distribution with expectation mu and deviation dev norm(x\mu,dev) = 1/(dev*sqrt(2*pi))*exp(-((x-mu)/(sqrt(2)*dev))*2) [0037] Let us denote by mu(X) an approximated expectation of a context X (X = A, B or C). It holds that: mu(A)=(-60-59-65)/3=-61.333; mu(B)=<-58-57-62)/3=-59; mu(C)=(-80,-85,-82,-81 ,-60)/5=-77.6; and dev(A)=dev(B)=dev(C)=5.

[0038] (The assumption of a constant deviation merely simplifies the example but does not in any way restrict the invention or its applications.) It holds now that empirical calibration approximations of the T-similarity measure are similarity(A→B|D(A))=2/3=0.666... , since norm(-60|mu(B),5)>norm(-60|mu(A),5), norm(-59|mu(B),5)>norm(-59|mu(A),5) and norm(-65|mu(B),5)<norm(-65|mu(A),5). [0039] This result, namely the similarity(A-»B|D(A))=2/3=67%, means that in two cases out of three, given the calibration data of the context A, a calibration observation gives a higher probability density for the wrong context B than for the correct context A.

[0041] Further, it is straightforward to verify that similarity(A→C|D(A))=0/3-0.0; similarity(C→A|D(C))=1/5=0.2; similarity(B→C|D(B))=0/3=0.0; and similarity(C→B|D(C))=1/5=0.2.

[0042] Reference numeral 26 denotes a T-similarity matrix for all similarity pairs between the contexts A, B and C. In the T-similarity matrix 26, cell (X₁Y) represents similarity(X-»Y|D(X)). T-similarity is a non-restrictive example of a similarity report.

[0043] Figure 3 visually illustrates how the measure similarity(A→B|D(A)) may be constructed, wherein reference numerafs 31 and 32 respectively denote context models for the two contexts A and B and reference numeral 33 denotes calibration data for context A. [0044] The numerical values in matrix 26 illustrate the above-mentioned fact that if context A is T-similar with context B, it does not necessarily follow that context B is T-similar with context A.

[0045] Given a T-similarity threshold value 0.25, it follows that the context A is T-similar with the context B and vice versa. This result suggests that the positioning system may mix up contexts A and B in view of the calibration data, and the user or operator should take some corrective action in order to improve the reliability of the positioning system (see the example explained in connection with Figure 2). For example, such corrective action may involve adding a new access point to context A and/or B. Alternatively or additionally an existing access point may be relocated, as will be explained in connection with Figures 5 through 9. Alternatively contexts A and B may be redefined by unifying them to one context, say D₁ since they are similar to one other. After the unification operation the environment consists of two contexts D and C that are not T-similar to one other. For example, in the example scenario related to the automatic lighting controlling system described earlier, keeping the lights turned on for both regions A and B is more acceptable than making a context estimation error letting the target object be in the dark. On the other hand, T- similarity measures indicate that given the current model, the context C will be reliably identified, since one outlier observation of -6OdBm in the calibration data of the context C does not dominate the calibration data. In general, small calibration data sets are more vulnerable to outliers in terms of T-similarity than bigger data sets. Accordingly, if similarities are present and the calibration data sets are small the corrective action may involve collecting more calibration data. However, in some cases contexts are so similar to each other that the context of a target object cannot be reliably estimated using a single observation made by a sensing device attached to the target object due to random variation in the observed values. For example, sequential radio signal strength observations may vary significantly even the radio receiver was static. In such cases, the corrective action may involve changing observing characteristics of the sensing device. For example, the sensing device may be configured to make multiple observations instead of one and provide a statistical summary of the observed values, such as an average or a median, to the context estimation system. [0046] A T-similarity matrix can be used to visually identify context pairs that are likely to be mixed up with one another, prior to seeing any test data. Ob- serve that T-similarity matrix is not the same object as the commonly known confusion matrix, wherein each column of the matrix represents a count of the instances in a predicted context, while each row represents a count of the instances in an actual context. A difference between the confusion matrix and T- similarity matrix is that each row X of the T-similarity matrix makes the closed- world assumption to construct the classifier using the score function of the context X and another score function of the given column Y (=context Y) to construct a local classifier. The confusion matrix, in turn, uses all score functions simultaneously in order to construct a global classifier. Consequently, if the classifier is modular, time complexity of the confusion matrix is 0(n*m), where n is the number of data points to be evaluated and m is the number of contexts, e.g. possible locations. Time complexity of the T-similarity matrix, on the other hand, is lower, namely O(n), which fact makes the T-similarity matrix faster to compute. This is particularly important when parameter values of the positioning system are optimized automatically using similarity between the contexts as an optimization criterion prior to seeing any test data. Another difference between the confusion matrix and the T-similarity matrix is that confusion matrix is constructed using the test data and T-similarity matrix may be constructed using calibration data only. [0047] Assume, for the present, that a prediction system needs to process n contexts C[1], C[2] C[n]. Let us introduce a set as follows:

SimilaritySet(C[i]) = {C[k] | context C[i] is T-similar with context C[k] }.

[0048] The SimilaritySet{C[Q) is the set of contexts that are similar with the context C[i] excluding itself. Automatic computation of the similarity sets is use- ful in identifying contexts that may be mixed up with one another, before test data is applied to the system. Another concrete example involves detecting zones that are similar with at least one other zone in terms of the positioning system and the calibration model. For instance, as regards the positioning application illustrated in Figure 2, given a threshold value T=0.25, it holds that SimilaritySet(A) = {B}, SimilaritySet(B) = {A} and SimilaritySet(C) = {}.

[0049] The similarity information can be utilized to improve robustness of a positioning system, by designing appropriate test survey paths in order to en- sure that the positioning system does not consider critical zones similar to one another. [0050] It is easy to derive global measures using local T-similarities. For instance, an average size of the similarity set serves as a compact global description of the simiiarities of the positioning model. For instance, in the example illustrated in Figure 2, the average size of a similarity set would be 2/3=0,66 indicating that 66% of the contexts are vulnerable.

[0051] Next, we outline a generalization of the T-similarity measure that can be used to assess noise tolerance of the prediction system. After that, we discuss how the general setting is applied in a statistical noise setting. [0052] Next a definition ((T,f,s)-similarity) will be given. Let f:S*O -> O be a mapping wherein S is a parametrization of f and O is the set of all possible data sets. Let s be an element of the parameter set S. Let 0<T<1 be a fixed real number. A context A is (T,f,s)-similar with a context B if a following inequality holds: similarity(A→B| f{s,D(A))=>T. [0053] That is, A is (T,f,s)-similar with B if and only if:

Pr[x is classified to context B | x is an element of the data set f(s,D(A)) > T.

[0054] Observe that if f is an identity mapping, then (T,f,s)-similarity reduces to T-similarity. (T,f,s)-simifarity is useful in analyzing phenomena that may arise in the actual use of the positioning system but did not occur during the calibration phase. For instance, it may occur that the signal values are over-dispersed or systematically lower for short periods of time. It is useful to know how the performance of the positioning system is affected during the short-term disorders. Performance of the positioning system can be assessed under the over- dispersion by defining the mapping f to be a random mapping such that it over- disperses the input data set via extra noise. The amount of expected noise is determined by the parameter s. If it turns out that A and B are (T,f,s)-srmilar, which is in this case a random variable, but they are not T-similar, then it is considered that A and B are potentially similar under exceptionally noisy conditions although they are not T-similar under normal conditions. It is possible to use statistical techniques, such as Monte Carlo -integration, to estimate the following probability:

Pr[Pr[X is classified to context B | x is an element of f(s,D(A))]>=T] (6).

[0055] Figure 4 shows a table which illustrates addition of noise to the example shown in Figure 2. As a concrete example, the data set shown in Figure 2 was generated by repeatedly taking a sample and adding extra noise to it. In this example, these acts were repeated 10,000 times. The noise was assumed additive and normally distributed with a variance of 9. For each noise-disturbed data set, T-similarity values were calculated, and expectations of the T- similarity values were computed via Monte Carlo integration. Reference numeral 41 denotes a data table which illustrates the corresponding expected T- similarity values. The table 41 shows that under the extra noise, T-similarity values of the context C are increased. However, none of the T-similarity values shown in table 41 exceeds the fixed threshold value T=0.25. This result sug- gests that positioning system is noise-tolerant as regards positioning in the context C given the threshold value T=0.25.

[0056] A concrete application example of the (T,f,s)-similarity will be presented in connection with Figures 5 through 9. A section of a WLAN network is illustrated in Figure 5. This exemplary section comprises two access points A and B, denoted by AP-A and AP-B, and two defined contexts C1 and C2. Calibration data collected in the contexts C1 and C2 is shown in Figure 6, where the values on the x-axis and y-axis respectively represent signal strengths of the access points A and B. The calibration data indicates that context C1 and C2 are not similar with one another. It is frequent in real-world positioning applica- tions, however, that for some reason signal strengths are weakened systematically. For instance, if there is an object that systematically blocks the signal path from the access points to a sensing device for a period of time, the observed signal strengths will be systematically lower than the signal strengths in the calibration data set. In order to simulate this phenomenon a mapping f is defined such that the mapping f systematically lowers the signal strengths. Figure 7 illustrates the corresponding manipulated calibration data sets of the contexts C1 and C2. Dashed regions, denoted by C1 and C2, represent the original high-density regions of the calibration data sets, while the two regions drawn in solid lines and denoted CV and C2\ represent the manipulated cali- bration data sets.

[0057] Figure 7 shows that after the systematic manipulation slightly more than half of the observations of the context C1 are classified to a wrong context C2 prior to seeing any test data. Consequently, there is a high risk that context C1 is too similar with context C2 when the signals are systematically blocked by an external object. In order to solve the problem, the layout of the WLAN network section can be changed. Figure 8 shows an example of a changed net- work layout, wherein the access point B, denoted by AP-B, has been moved to the lower right-hand corner of the network section. Figure 9 shows both the original distributions of the calibration data set values and systematically manipulated (biased) data distributions for the contexts C1 and C2 given the net- work layout shown in Figure 8. Figure 9 shows that even after systematic manipulation, the contexts C1 and C2 are not mixed up with one another. This result suggests that the network layout shown in Figure 8 is robust with respect to the systematic signal blocking phenomenon. [0058] In the following, Monte Carlo integration will be further discussed since it is a useful technique for some applications which are used to implement the invention, in addition to the previous examples. In a Monte Carlo integration, let X be an arbitrary binary random variable that has two possible realizations, say 0 and 1. Suppose it is possible to generate realizations from X's probability distribution. Let {x[1 ],..., x[n]} be independent and identically distributed random sample from X's distribution, where x[i]=0 or x[i]=1 for all ϊ=1 ,2, n. Probability of the event that X=O can now be approximated as

Pr[X=O] ~ (l(x[1]=0)+l(x[2]=0)+...+l(x[n]=0)) / n. [6]

[0059] Herein l[.] is an indicator function that gets a value 1 whenever the input is true and value 0 otherwise. This technique is known as Monte Carlo integra- tion approximation and it has theoretical guarantees in a sense that the approximated probability converges in probability to the true probability value. [0060] An exemplary implementation is as follows. In order to solve the above- described problem of false context identification, the positioning system may approximate the following probability: Pr[x is classified to context B|x was collected in the context A]. [7]

[0061] Those skilled in the art will realize that it is possible to approximate the probability in equation [4] by applying the Monte Carlo integration procedure described above. First, each observation should be evaluated regarding whether or not it is miss-classified. Second, the desired probability in equation [1] can be approximated as a proportion of observations that were miss- classified. If the approximated probability exceeds the given threshold T, ie, the proportion of observations that were miss-classified, it is considered that the context A is T-similar with the context B. [0062] Observe that evaluation of the conditional marginal likelihoods Pr(x|A) and Pr(x|B) (see equation [3]) is often computationally intractable. As a conse- quence, maximum likelihood parameter values may be adopted. We denote by M(X) the maximum likelihood parameter values of the context X=A₁B. Equation (3) now obtains a computationally more tractable form given a uniform prior over the contexts Pr(A|x) = Pr(x|A,M(A))/Z. [8]

[0063] Herein Z=(Pr(x|A,M(A))+Pr(x|B,M(B))) is a normalization constant. [0064] If the T-similarity measure is to be used to assess noise-tolerance, the procedure is almost similar to the one just described, but extra noise defined by variable s may be added to the calibration data D(A). An alternative is to apply random re-sampling techniques but they may exaggerate the value of D(A).

[0065] The inventive technique can be used to speed up the process of constructing a reliable positioning model for a positioning system or to reduce resource consumption in such a model-construction building process. The inven- tion also provides an opportunity to assess the noise tolerance of the system prior to applying any test data to the system. Although the inventive technique can utilize the test data, it doesn't necessarily need any. The inventive technique can also be used to automatically detect ambiguous contexts, which is beneficial in critical positioning applications. Yet further, it can be utilized in designing test case survey routes, since it identifies automatically potential error-prone regions without requiring any actual test data; actual test data may be used to verify how severe the ambiguity is. In fact, if the test data is similar to calibration data, it is likely that big errors will occur in the occasional positioning task in the identified ambiguous regions. [0066] The inventive technique can be used to assess and/or improve accuracy of a tracking-based positioning algorithm as follows. Suppose a tracking positioning algorithm is based on a prediction of a region where the target object is moving. When it is likely that the target object is approaching a region that has at least one T-similar region, parameters of the positioning algorithm can be adjusted in order to take into account similarity such that undesired jumps will be less likely. For instance, in a positioning technique which involves probability particles which undergo a stochastic evolutionary process (see reference document 5), the evolutionary process could be adjusted as follows. If it is likely that the target object is approaching a T-similar region, then the pro- portion of the probability particles that are generated independently of the pre- vious generation is decreased to zero or almost zero. This adjustment makes it unlikely that a new particle generation will be born in a wrong T-similar region. [0067J The inventive technique can be used in dimension reduction. One non- restricting example of the dimension reduction is to reduce the number of ob- servable context-dependent physical quantities used in the context models as long as the context estimation system does not have similar contexts. For example, signal strength observations for a certain radio transmitter may be omitted from calculations in case this does not decrease context estimation system performance below acceptable level. Another example of reducing the number of quantities is combining two or more quantities by replacing the values of the combined quantities with a single value derived from the values of the combined quantities using some mathematical function. Another non-restricting example of the dimension reduction is to reduce the number of contexts by unifying similar contexts. Benefits of the dimension reduction, in turn, include faster computation, data compression and new descriptive information regarding the problem domain.

[0068] The inventive technique can be used to automatically determine a reliable granularity of a context estimation system. One non-restricting example is the previously described dimension reduction via context unification. Another non-restricting example is to split a context to at least two new contexts. If the new contexts are not similar with each other it is an indication that a context estimation system is capable to model more detailed context features than specified in the first place.

[0069] The inventive technique can be used to in model selection. One non- restricting example of the model selection is as follows. Assume there are three different context model types that are based on Bayesian networks, neural networks and decision trees. For each context model type T-similarities are evaluated using a context estimation system and the context model type that is the most suitable in terms of the similarities is chosen to be used in the final context estimation system.

[0070] In principle, the inventive method can be used to detect when the calibration model is obsolete locally vs. globally, for example by using so-called reference target devices or by collecting a new test case from time to time. If the calibration model is obsolete, then the latest data collected by the refer- ence device is not T-similar with the original local calibration data, and a new calibration effort is required either via reference devices or by manual calibra- tion. This is useful, in particular, if the positioning system is modular in the sense that local models can be updated independently of one another. [0071] Although specific embodiments and implementation examples were described to facilitate understanding the invention, those skilled in the art will realize that the invention is not restricted to the specific embodiments and implementation details. For example, data derived by computer simulations, such as ray-tracing techniques, may be used in constructing the statistical model instead of actually measured calibration data or in addition to it.

Reference documents: 1. WO2004/008795 discloses location-determination techniques which use a graph that models the topology of the target object's communication environment.

2. WO03/102622 discloses techniques for locating a target in a wireless environment. The techniques use a plurality of submodels of the wireless envi- ronment, each submodel indicating a probability distribution for signal values at one or more locations in the wireless environment. The submodels are combined to a probabilistic model of the environment which indicates probability distributions for signal values in the environment.

3. WO2004/008796 discloses a location-determination technique which com- prises determining a plurality of device models that compensate for the differences between different target objects' observations of signal quality parameters and selecting, among the multiple device models, a specific device model for a specific target object.

4. WO02/054813 discloses methods and equipment for estimating a re- ceiver's location in a wireless telecommunication environment.

5. Finnish patent application FI20055649 (unpublished at the priority date of the present application, later published as EP1796419A) discloses a method and system for estimating a target object's properties, including location, in an environment. A topology model indicates permissible locations and transitions and a data model models a location-dependent physical quantity which is observed by the target object's co-located sensing device. Motion models model specific target object types, obeying the permissible locations and transitions. The target object is assigned a set of particles, each having a set of attributes, including location in relation to the topology model. The attributes estimate the target object properties. The particles' update cycles comprise: determining a degree of belief for each particle to estimate the target object properties; determining a weight for each particle based on at least the determined degree of belief and generating new particles for update cycle n+1 in an evolutionary process.

[0072] The above reference documents, which are incorporated herein by ref- erence, are commonly-owned patent appfications describing illustrative but non-restrictive positioning techniques compatible with the present invention.

Claims

1. A method for generating a computer-implemented context estimation system for an environment including at least one observable context-dependent physical quantity, wherein the context estimation system comprises a data proces- sor and memory and is capable of outputting a context estimate for a target object using an observation made by a sensing device attached to the target object, wherein the observation comprises at least one value for one or more physical quantities, and wherein the context estimate is capable of being used by a context-aware application to trigger one or more context-related actions, the method comprising: a) defining two or more contexts (11 , 22, 23, 24, C1 , C2), wherein each context defines the circumstances under which observations of the physical quantities are made; b) providing calibration data (25) for each context, wherein the calibration data for a context comprises several observations, each of which is made under the circumstances defined by the context and comprises values for one or more physical quantities; c) constructing a context model (31 , 32) for each context using the calibration data of the context, wherein the context models are used by a context score function taking an observation and a context as input and returning a score which indicates a likelihood that the inputted observation is made under the inputted context; d) generating a similarity report (13) identifying similarities between context models, wherein said generating the similarity report comprises: • selecting a first context and a second context;

• selecting an observation from the calibration data of the first context;

• calculating a second score using the context score function giving the selected observation and the second context as input; and

• updating the similarity report using the first score and the second score; e) making changes affecting the performance of the context estimation system by utilizing the similarity report; and f) storing the context models and the context score function in the memory of the context estimation system, whereby the context estimation system is capable of outputting a context estimate for the target object, wherein the context estimate is determined using the scores returned by the context score function when given an observation made by a sensing device attached to the target object as input.

2. The method according to claim 1 , wherein the circumstances defined by a context comprise specification of locations within the environment at which observations for the context may be made,

3. The method according to claim 1 , wherein at least one of the physical quantities is based on radio signals transmitted by a radio transmitter at a fixed lo- cation.

4. The method according to claim 1 , wherein the step of making changes affecting the performance of the context estimation system comprises redefining the contexts.

5. The method according to claim 1 , wherein the step of making changes af- fecting the performance of the context estimation system comprises providing more calibration data.

6. The method according to claim 1 , wherein the step of making changes affecting the performance of the context estimation system comprises reconstructing the context models using an alternative context model type.

7. The method according to claim 3, wherein the step of making changes affecting the performance of the context estimation system comprises moving a radio transmitter related to at least one of the physical quantities to another location.

8. The method according to claim 1 , wherein the step of making changes af- fecting the performance of the context estimation system comprises increasing the number of observable context-dependent physical quantities within the environment.

9. The method according to claim 1 , wherein the step of making changes affecting the performance of the context estimation system comprises changing observing characteristics of the sensing device attached to the target object.

10. The method according to claim 1 , wherein the step of generating the similarity report further comprises manipulating observations selected from the calibration data of the first context before calculating the first score and the second score.

11. The method according to claim 10, wherein the step of manipulating observations comprises adding noise to at least one of the physical quantity values.

12. The method according to claim 10, wherein the step of specifying performance requirements for the context estimation system comprises simulating one or more phenomena which affect observations of one or more physical quanti- ties.

13. The method according to claim 1 , wherein the step of making changes affecting the performance of the context estimation system comprises omitting one ore more physical quantities from calculations in order to reduce computational complexity of the context estimation system.

14. The method according to claim 1 , wherein the performance of the context estimation system is improved in a recursive process utilizing similarity reports in order to satisfy predetermined performance requirements, the method further comprising: g) specifying performance requirements for the context estimation system; h) estimating performance of the context estimation system using the similarity report; and i) repeating steps d), e), and h) until the estimated performance of the context estimation system satisfies the performance requirements.

15. A set of computer-readable media for a data processing system which comprises a model construction section and a context estimation section, wherein the set of computer-readable media comprises:

- first computer program instructions whose execution in the model construction section causes the model construction section to carry out steps a) through d) of claim 1 ; and - second computer program instructions whose execution in the context estimation section causes the context estimation section to carry out step f) according to claim 1.

16. A data processing system which comprises a model construction section and a context estimation section, wherein the data processing system further comprises the set of computer-readable media according to claim 15.