Distance Functions on Hierarchies

Παρουσίαση με θέμα: "Distance Functions on Hierarchies"— Μεταγράφημα παρουσίασης:

1 Distance Functions on Hierarchies
Eftychia Baikousi

2 Outline Definition of metric & similarity Various Distance Functions
Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

3 Definition of metric A distance function on a given set M is a function d:MxM , that satisfies the following conditions: d(x,y)≥0 and d(x,y)=0 iff x=y Distance is positive between two different points and is zero precisely from a point to itself It is symmetric: d(x,y)=d(y,x) The distance between x and y is the same in either direction It satisfies the triangle inequality: d(x,z) ≤ d(x,y)+ d(y,z) The distance between two points is the shortest distance along any path Is a metric

4 Definition of similarity metric
Let s(x,y) be the similarity between two points x and y, then the following properties hold: s(x,y) =1 only if x=y (0≤ s ≤1) s(x,y) =s(y,x)  x and y (symmetry) The triangle inequality does not hold

5 Outline Various Distance Functions Definition of metric & similarity
Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

6 Minkowski Family L1(x,y)= Σi |xi-yi| L2(x,y)=(Σi |xi-yi|2 )1/2
norm-1, City-Block, Manhattan L1(x,y)= Σi |xi-yi| norm-2, Euclidian L2(x,y)=(Σi |xi-yi|2 )1/2 norm-p, Minkowski Lp(x,y)=(Σi |xi-yi|p )1/p infinity norm L=limp  (Σi |xi-yi|p )1/p =maxi (|xi-yi|)

7 Set Based Simple matching coefficient Jaccard Coefficient
Extended Jaccard, Tanimoto (Vector based) Cosine (Vector based) Dice’s coefficient

8 Edit Distance- Levenshtein distance
Edit distance between two strings x=x1 ….xn, y=y1…ym is defined as the minimum number of atomic edit operations needed Insert : ins(x,i,c)=x1x2…xicxi+1…xn Delete : del(x,i)=x1x2…xi-1xi+1…xn Replace : rep(x,i,c)=x1x2…xi-1cxi+1…xn Assign cost for every edit operation c(o)=1

9 Edit distances Needleman-Wunch distance or Sellers Algorithm Insert
a character ins(x,i,c)=x1x2…xicxi+1…xn with cost(o)=1 a gap ins_g(x,i,g)=x1x2…xigxi+1…xn with cost(o)=g Delete a character del(x,i)=x1x2…xi-1xi+1…xn a gap del_g(x,i)=x1x2…xi-1xi+1…xn Replace a character rep(x,i,c)=x1x2…xi-1cxi+1…xn

10 Edit distances Jaro distance Let two strings s and t and
s’= characters in s that are common with t t’ = characters in t that are common with s Ts,t =number of transportations of characters in s’ relative to t’

11 Edit distances Jaro distance Example Let s =MARTHA and t =MARHTA
Ts,t = 2/2 since mismatched characters are T/H and H/T

12 Edit distances Jaro Winkler
JWS(s,t)= Jaro(s,t) + ((prefixLength * PREFIXSCALE * (1.0-Jaro(s,t))) Where: prefixLength : the length of common prefix at the start of the string PREFIXSCALE: a constant scaling factor which gives more favourable ratings to strings that match from the beginning for a set prefix length

13 Edit distances Jaro Winkler Example
Let s =MARTHA and t =MARHTA and PREFIXSCALE = 0.1 Jaro(s,t)=0.8055 prefixLength=3 JWS(s,t)= Jaro(s,t) + ((prefixLength * PREFIXSCALE * (1.0-Jaro(s,t))) = (3*0.1*( )) =

14 Outline Basic concept of OLAP Definition of metric & similarity
Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

15 Βασικές Έννοιες OLAP Αφορά την ανάλυση κάποιων μετρήσιμων μεγεθών (μέτρων) πωλήσεις, απόθεμα, κέρδος,... Διαστάσεις: παράμετροι που καθορίζουν το περιβάλλον (context) των μέτρων ημερομηνία, προϊόν, τοποθεσία, πωλητής, … Κύβοι: συνδυασμοί διαστάσεων που καθορίζουν κάποια μέτρα Ο κύβος καθορίζει ένα πολυδιάστατο χώρο διαστάσεων, με τα μέτρα να είναι σημεία του χώρου αυτού

16 Κύβοι για OLAP REGION N S W PRODUCT Juice Cola Soap MONTH Jan 10 13

17 Κύβοι για OLAP

18 Βασικές Έννοιες OLAP Τα δεδομένα θεωρούνται αποθηκευμένα σε ένα πολυδιάστατο πίνακα (multi-dimensional array), ο οποίος αποκαλείται και κύβος ή υπερκύβος (Cube και HyperCube αντίστοιχα). Ο κύβος είναι μια ομάδα από κελιά δεδομένων (data cells). Κάθε κελί χαρακτηρίζεται μονοσήμαντα από τις αντίστοιχες τιμές των διαστάσεων (dimensions) του κύβου. Τα περιεχόμενα του κελιού ονομάζονται μέτρα (measures) και αναπαριστούν τις αποτιμώμενες αξίες του πραγματικού κόσμου.

19 Ιεραρχίες επιπέδων για OLAP
Μια διάσταση μοντελοποιεί όλους τους τρόπους με τους οποίους τα δεδομένα μπορούν να συναθροιστούν σε σχέση με μια συγκεκριμένη παράμετρο του περιεχομένου τους. Ημερομηνία, Προϊόν, Τοποθεσία, Πωλητής, … Κάθε διάσταση έχει μια σχετική ιεραρχία επιπέδων συνάθροισης των δεδομένων (hierarchy of levels). Αυτό σημαίνει, ότι η διάσταση μπορεί να θεωρηθεί από πολλά επίπεδα αδρομέρειας. Ημερομηνία: μέρα, εβδομάδα, μήνας, χρόνος, …

20 Ιεραρχίες Επιπέδων Ιεραρχίες Επιπέδων: κάθε διάσταση οργανώνεται σε διαφορετικά επίπεδα αδρομέρειας Ο χρήστης μπορεί να πλοηγηθεί από το ένα επίπεδο στο άλλο, δημιουργώντας νέους κύβους κάθε φορά Αδρομέρεια: το αντίθετο της λεπτομέρειας -- ο σωστός όρος είναι αδρομέρεια...

21 Κύβοι & ιεραρχίες διαστάσεων για OLAP
Month Region Product Sales volume Διαστάσεις: Product, Region, Date Ιεραρχίες διαστάσεων: Industry Country Year Category Region Quarter Product City Month Week Store Day

22 Outline Lattice Definition of metric & similarity
Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

23 Lattice A lattice is a partially ordered set (poset) in which every pair of elements has a unique supremum and an inifimum The hierarchy of levels is formally defined as a lattice (L,<) such that L= (L1, ..., Ln, ALL) is a finite set of levels and < is a partial order defined among the levels of L such that L1<Li<ALL  1≤i≤n. the upper bound is always the level ALL, so that we can group all values into the single value ‘all’. The lower bound of the lattice is the most detailed level of the dimension.

24 Outline Distance in same level of hierarchy
Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

25 Distances in the same level of Hierarchy
Let a dimension D, its levels of hierarchies L1<Li<ALL and two specific values x and y s.t. x, y  Li All L2 L1

26 Distances in the same level of Hierarchy
Explicit Based on instance Based on property Minkowski Path WRT to descendant WRT to ancestor Highway

27 Explicit assignment (Instance)
All n2 distances for the n values of the dom(Li) Identity Example dist (x, y) = dist (milk, milk)= 0 dist (x, y) = dist (milk, yogurt)=1 Li x1 x2 …… xn All Products milk yogurt …… cola

28 Explicit assignment (Property)
Attribute based  level L  attributes:  v [v1 … vn]  dom(L) Distance defined wrt attributes Example Let the level Product have attributes: [color, weight] dist (x, y)= dist (milk, cola)= dist (color(milk), color(cola))= dist (white, black) All Products milk yogurt …… cola

29 Explicit assignment (Property)
Applying ancestor function Let x, y  Li then Examples dist (milk, yogurt) =0 dist (milk, cola)=1 Applying descendant relation All Type dairy Soft drink Product milk yogurt cola orange

30 Minkowski family reduce to Manhattan distance: |x-y| Example
dist (x, y)=dist( ) = | |=10 All Year 1990 1995 2000 2005

31 Path distances With respect to descendants Lower level
Let x and y  Li and Lj a lower level , i.e., Lj < Li f : a function that picks one of the descendants Special case when Lj is the detailed level L1

32 Path distances WRT detailed level Example
dist (Greece, France)= dist (Athens, Paris) Where let All Country Greece France City Ioannina Athens Paris Nantes

33 Path distances Example With respect to the least common ancestor
Let x and y  Li and Lj the upper level, i.e., Lj < Li where x and y have a common ancestor, Example dist (x, y)= dist (Athens, Paris)= dist (Athens, Europe) + +dist (Europe, Paris) = 2 + 2 = 4 All Continent Europe Country Greece France City Ioannina Athens Paris Nantes

34 Highway distance dist(x, y)= dist(x, rx)+ dist(rx, ry)+ dist(y, ry)
Let the values  Li form a set of k clusters, each cluster has a representative rk dist(x, y)= dist(x, rx)+ dist(rx, ry)+ dist(y, ry) Specify: k2 distances: dist (rx, ry) and k distances: dist(x, rx) Example: dist (Ioannina, Nantes)= dist (Ioannina, Athens) + dist (Athens, Paris) + dist (Paris, Nantes) All Country Greece France City Ioannina Athens Paris Nantes

35 Outline Distance in different level of hierarchy
Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

36 Distances in different levels of Hierarchy
Explicit dist1+ dist2 dist3+dist4 WRT the detailed level WRT their least common ancestor Highway Attribute Based

37 Distances in different levels of Hierarchy
Let a dimension D, its levels of hierarchies L1<Li<ALL two specific values x and y s. t. x Lx  y Ly Lx<Ly ancestor of x in level Ly a descendant of y in level Lx yx xy Ly x y dist1 dist3 dist2 dist4 Lx

38 Distances in different levels of Hierarchy
yx xy Ly x y dist1 dist3 dist2 dist4 Lx Explicit assignment define distLx,Ly(x, y) x  Lx, y Ly dist1 +dist2 Where is a distance of two values from the same level of hierarchy special case: y is an ancestor of x then dist2=0

39 Distances in different levels of Hierarchy
yx xy Ly x y dist1 dist3 dist2 dist4 Lx dist3 +dist4 f : function that picks a descendant Where a distance of two values from the same level of hierarchy special case: y is an ancestor of x then dist4=0

40 Distances in different levels of Hierarchy
With respect to the detailed level Let and Where dist(x1, y1) a distance of two values from the same level of hierarchy

41 Distances in different levels of Hierarchy
With respect to their common ancestor Let Lz the level of hierarchy where x and y have their first common ancestor number of “hops” needed to reach the first common ancestor normalizing according to the height of the level

42 Distances in different levels of Hierarchy
Highway distance Let every Li is clustered into ki clusters and every cluster has its own representative rki Attribute Based  level L  attributes:  v [v1 … vn]  dom(L) Distance can be defined with respect to the attributes

43 Types of Levels Nominal =  Ordinal < > Interval + -
values hold the distinctness property values can be explicitly distinguished Ordinal < > values hold the distinctness property & the order property values abide by an order Interval + - values hold the distinctness, order & the addition property a unit of measurement exists there is meaning of the difference between two values