Distance Functions on Hierarchies Eftychia Baikousi

Outline Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

Definition of metric A distance function on a given set M is a function d:MxM , that satisfies the following conditions: d(x,y)≥0 and d(x,y)=0 iff x=y Distance is positive between two different points and is zero precisely from a point to itself It is symmetric: d(x,y)=d(y,x) The distance between x and y is the same in either direction It satisfies the triangle inequality: d(x,z) ≤ d(x,y)+ d(y,z) The distance between two points is the shortest distance along any path Is a metric

Definition of similarity metric Let s(x,y) be the similarity between two points x and y, then the following properties hold: s(x,y) =1 only if x=y (0≤ s ≤1) s(x,y) =s(y,x)  x and y (symmetry) The triangle inequality does not hold

Outline Various Distance Functions Definition of metric & similarity Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

Minkowski Family L1(x,y)= Σi |xi-yi| L2(x,y)=(Σi |xi-yi|2 )1/2 norm-1, City-Block, Manhattan L1(x,y)= Σi |xi-yi| norm-2, Euclidian L2(x,y)=(Σi |xi-yi|2 )1/2 norm-p, Minkowski Lp(x,y)=(Σi |xi-yi|p )1/p infinity norm L=limp  (Σi |xi-yi|p )1/p =maxi (|xi-yi|)

Set Based Simple matching coefficient Jaccard Coefficient Extended Jaccard, Tanimoto (Vector based) Cosine (Vector based) Dice’s coefficient

Edit Distance- Levenshtein distance Edit distance between two strings x=x1 ….xn, y=y1…ym is defined as the minimum number of atomic edit operations needed Insert : ins(x,i,c)=x1x2…xicxi+1…xn Delete : del(x,i)=x1x2…xi-1xi+1…xn Replace : rep(x,i,c)=x1x2…xi-1cxi+1…xn Assign cost for every edit operation c(o)=1

Edit distances Needleman-Wunch distance or Sellers Algorithm Insert a character ins(x,i,c)=x1x2…xicxi+1…xn with cost(o)=1 a gap ins_g(x,i,g)=x1x2…xigxi+1…xn with cost(o)=g Delete a character del(x,i)=x1x2…xi-1xi+1…xn a gap del_g(x,i)=x1x2…xi-1xi+1…xn Replace a character rep(x,i,c)=x1x2…xi-1cxi+1…xn

Edit distances Jaro distance Let two strings s and t and s’= characters in s that are common with t t’ = characters in t that are common with s Ts,t =number of transportations of characters in s’ relative to t’

Edit distances Jaro distance Example Let s =MARTHA and t =MARHTA Ts,t = 2/2 since mismatched characters are T/H and H/T

Edit distances Jaro Winkler JWS(s,t)= Jaro(s,t) + ((prefixLength * PREFIXSCALE * (1.0-Jaro(s,t))) Where: prefixLength : the length of common prefix at the start of the string PREFIXSCALE: a constant scaling factor which gives more favourable ratings to strings that match from the beginning for a set prefix length

Edit distances Jaro Winkler Example Let s =MARTHA and t =MARHTA and PREFIXSCALE = 0.1 Jaro(s,t)=0.8055 prefixLength=3 JWS(s,t)= Jaro(s,t) + ((prefixLength * PREFIXSCALE * (1.0-Jaro(s,t))) = 0.8055 + (3*0.1*(1-0.8055)) = 0.86385

Outline Basic concept of OLAP Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

Βασικές Έννοιες OLAP Αφορά την ανάλυση κάποιων μετρήσιμων μεγεθών (μέτρων) πωλήσεις, απόθεμα, κέρδος,... Διαστάσεις: παράμετροι που καθορίζουν το περιβάλλον (context) των μέτρων ημερομηνία, προϊόν, τοποθεσία, πωλητής, … Κύβοι: συνδυασμοί διαστάσεων που καθορίζουν κάποια μέτρα Ο κύβος καθορίζει ένα πολυδιάστατο χώρο διαστάσεων, με τα μέτρα να είναι σημεία του χώρου αυτού

Κύβοι για OLAP REGION N S W PRODUCT Juice Cola Soap MONTH Jan 10 13

Κύβοι για OLAP

Βασικές Έννοιες OLAP Τα δεδομένα θεωρούνται αποθηκευμένα σε ένα πολυδιάστατο πίνακα (multi-dimensional array), ο οποίος αποκαλείται και κύβος ή υπερκύβος (Cube και HyperCube αντίστοιχα). Ο κύβος είναι μια ομάδα από κελιά δεδομένων (data cells). Κάθε κελί χαρακτηρίζεται μονοσήμαντα από τις αντίστοιχες τιμές των διαστάσεων (dimensions) του κύβου. Τα περιεχόμενα του κελιού ονομάζονται μέτρα (measures) και αναπαριστούν τις αποτιμώμενες αξίες του πραγματικού κόσμου.

Ιεραρχίες επιπέδων για OLAP Μια διάσταση μοντελοποιεί όλους τους τρόπους με τους οποίους τα δεδομένα μπορούν να συναθροιστούν σε σχέση με μια συγκεκριμένη παράμετρο του περιεχομένου τους. Ημερομηνία, Προϊόν, Τοποθεσία, Πωλητής, … Κάθε διάσταση έχει μια σχετική ιεραρχία επιπέδων συνάθροισης των δεδομένων (hierarchy of levels). Αυτό σημαίνει, ότι η διάσταση μπορεί να θεωρηθεί από πολλά επίπεδα αδρομέρειας. Ημερομηνία: μέρα, εβδομάδα, μήνας, χρόνος, …

Ιεραρχίες Επιπέδων Ιεραρχίες Επιπέδων: κάθε διάσταση οργανώνεται σε διαφορετικά επίπεδα αδρομέρειας Ο χρήστης μπορεί να πλοηγηθεί από το ένα επίπεδο στο άλλο, δημιουργώντας νέους κύβους κάθε φορά Αδρομέρεια: το αντίθετο της λεπτομέρειας -- ο σωστός όρος είναι αδρομέρεια...

Κύβοι & ιεραρχίες διαστάσεων για OLAP Month Region Product Sales volume Διαστάσεις: Product, Region, Date Ιεραρχίες διαστάσεων: Industry Country Year Category Region Quarter Product City Month Week Store Day

Outline Lattice Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

Lattice A lattice is a partially ordered set (poset) in which every pair of elements has a unique supremum and an inifimum The hierarchy of levels is formally defined as a lattice (L,<) such that L= (L1, ..., Ln, ALL) is a finite set of levels and < is a partial order defined among the levels of L such that L1<Li<ALL  1≤i≤n. the upper bound is always the level ALL, so that we can group all values into the single value ‘all’. The lower bound of the lattice is the most detailed level of the dimension.

Outline Distance in same level of hierarchy Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

Distances in the same level of Hierarchy Let a dimension D, its levels of hierarchies L1<Li<ALL and two specific values x and y s.t. x, y  Li All L2 L1

Distances in the same level of Hierarchy Explicit Based on instance Based on property Minkowski Path WRT to descendant WRT to ancestor Highway

Explicit assignment (Instance) All n2 distances for the n values of the dom(Li) Identity Example dist (x, y) = dist (milk, milk)= 0 dist (x, y) = dist (milk, yogurt)=1 Li x1 x2 …… xn All Products milk yogurt …… cola

Explicit assignment (Property) Attribute based  level L  attributes:  v [v1 … vn]  dom(L) Distance defined wrt attributes Example Let the level Product have attributes: [color, weight] dist (x, y)= dist (milk, cola)= dist (color(milk), color(cola))= dist (white, black) All Products milk yogurt …… cola

Explicit assignment (Property) Applying ancestor function Let x, y  Li then Examples dist (milk, yogurt) =0 dist (milk, cola)=1 Applying descendant relation All Type dairy Soft drink Product milk yogurt cola orange

Minkowski family reduce to Manhattan distance: |x-y| Example dist (x, y)=dist(1995-2005) = |1995-2005|=10 All Year 1990 1995 2000 2005

Path distances With respect to descendants Lower level Let x and y  Li and Lj a lower level , i.e., Lj < Li f : a function that picks one of the descendants Special case when Lj is the detailed level L1

Path distances WRT detailed level Example dist (Greece, France)= dist (Athens, Paris) Where let All Country Greece France City Ioannina Athens Paris Nantes

Path distances Example With respect to the least common ancestor Let x and y  Li and Lj the upper level, i.e., Lj < Li where x and y have a common ancestor, Example dist (x, y)= dist (Athens, Paris)= dist (Athens, Europe) + +dist (Europe, Paris) = 2 + 2 = 4 All Continent Europe Country Greece France City Ioannina Athens Paris Nantes

Highway distance dist(x, y)= dist(x, rx)+ dist(rx, ry)+ dist(y, ry) Let the values  Li form a set of k clusters, each cluster has a representative rk dist(x, y)= dist(x, rx)+ dist(rx, ry)+ dist(y, ry) Specify: k2 distances: dist (rx, ry) and k distances: dist(x, rx) Example: dist (Ioannina, Nantes)= dist (Ioannina, Athens) + dist (Athens, Paris) + dist (Paris, Nantes) All Country Greece France City Ioannina Athens Paris Nantes

Outline Distance in different level of hierarchy Definition of metric & similarity Various Distance Functions Minkowski Set based Edit distance Basic concept of OLAP Lattice Distance in same level of hierarchy Distance in different level of hierarchy

Distances in different levels of Hierarchy Explicit dist1+ dist2 dist3+dist4 WRT the detailed level WRT their least common ancestor Highway Attribute Based

Distances in different levels of Hierarchy Let a dimension D, its levels of hierarchies L1<Li<ALL two specific values x and y s. t. x Lx  y Ly Lx<Ly ancestor of x in level Ly a descendant of y in level Lx yx xy Ly x y dist1 dist3 dist2 dist4 Lx

Distances in different levels of Hierarchy yx xy Ly x y dist1 dist3 dist2 dist4 Lx Explicit assignment define distLx,Ly(x, y) x  Lx, y Ly dist1 +dist2 Where is a distance of two values from the same level of hierarchy special case: y is an ancestor of x then dist2=0

Distances in different levels of Hierarchy yx xy Ly x y dist1 dist3 dist2 dist4 Lx dist3 +dist4 f : function that picks a descendant Where a distance of two values from the same level of hierarchy special case: y is an ancestor of x then dist4=0

Distances in different levels of Hierarchy With respect to the detailed level Let and Where dist(x1, y1) a distance of two values from the same level of hierarchy

Distances in different levels of Hierarchy With respect to their common ancestor Let Lz the level of hierarchy where x and y have their first common ancestor number of “hops” needed to reach the first common ancestor normalizing according to the height of the level

Distances in different levels of Hierarchy Highway distance Let every Li is clustered into ki clusters and every cluster has its own representative rki Attribute Based  level L  attributes:  v [v1 … vn]  dom(L) Distance can be defined with respect to the attributes

Types of Levels Nominal =  Ordinal < > Interval + - values hold the distinctness property values can be explicitly distinguished Ordinal < > values hold the distinctness property & the order property values abide by an order Interval + - values hold the distinctness, order & the addition property a unit of measurement exists there is meaning of the difference between two values