TAPER: A two-step approach for all-strong-pairs correlation query in large databases

Hui Xiong; Shashi Shekhar; Pang Ning Tan; Vipin Kumar

doi:10.1109/TKDE.2006.1599388

TAPER: A two-step approach for all-strong-pairs correlation query in large databases

Hui Xiong, Shashi Shekhar, Pang Ning Tan, Vipin Kumar

Computer Science and Engineering

Research output: Contribution to journal › Article › peer-review

38 Scopus citations

Abstract

Given a user-specified minimum correlation threshold θ and a market-basket database with N items and T transactions, an all-strong-pairs correlation query finds all item pairs with correlations above the threshold θ. However, when the number of items and transactions are large, the computation cost of this query can be very high. The goal of this paper is to provide computationally efficient algorithms to answer the all-strong-pairs correlation query. Indeed, we identify an upper bound of Pearson's correlation coefficient for binary variables. This upper bound is not only much cheaper to compute than Pearson's correlation coefficient, but also exhibits special monotone properties which allow pruning of many item pairs even without computing their upper bounds. A Two-step All-strong-Pairs corElation queRy (TAPER) algorithm is proposed to exploit these properties in a filter-and-refine manner. Furthermore, we provide an algebraic cost model which shows that the computation savings from pruning is independent of or improves when the number of items is increased in data sets with Zipf-like or linear rank-support distributions. Experimental results from synthetic and real-world data sets exhibit similar trends and show that the TAPER algorithm can be an order of magnitude faster than brute-force alternatives. Finally, we demonstrate that the algorithmic ideas developed in the TAPER algorithm can be extended to efficiently compute negative correlation and uncentered Pearson's correlation coefficient.

Original language	English (US)
Pages (from-to)	493-508
Number of pages	16
Journal	IEEE Transactions on Knowledge and Data Engineering
Volume	18
Issue number	4
DOIs	https://doi.org/10.1109/TKDE.2006.1599388
State	Published - Apr 2006

Bibliographical note

Funding Information:
This work was partially supported by US National Science Foundation (NSF) grant IIS-0308264, NSF grant ITR ACI-0325949, and by the US Army High Performance Computing Research Center under the auspices of the Department of the Army, Army Research Laboratory cooperative agreement number DAAD19-01-2-0014. The content of this work does not necessarily reflect the position or policy of the government and no official endorsement should be inferred. Access to computing facilities was provided by the AHPCRC and the Minnesota Supercomputing Institute. In addition, the authors thank Professor Joydeep Ghosh (University of Texas, Austin) and Professor Ke Wang (Simon Fraser University, Canada) for valuable comments. Finally, they are grateful to Kim Koffolt for her timely and detailed feedback to help improve the readability of this paper.

Keywords

Association analysis
Data mining
Pearson's correlation coefficient
Statistical computing

Access

10.1109/TKDE.2006.1599388

OpenUrl availability

Full text

Cite this

@article{dbeaec006362492c9aa73371ccf66db6,

title = "TAPER: A two-step approach for all-strong-pairs correlation query in large databases",

abstract = "Given a user-specified minimum correlation threshold θ and a market-basket database with N items and T transactions, an all-strong-pairs correlation query finds all item pairs with correlations above the threshold θ. However, when the number of items and transactions are large, the computation cost of this query can be very high. The goal of this paper is to provide computationally efficient algorithms to answer the all-strong-pairs correlation query. Indeed, we identify an upper bound of Pearson's correlation coefficient for binary variables. This upper bound is not only much cheaper to compute than Pearson's correlation coefficient, but also exhibits special monotone properties which allow pruning of many item pairs even without computing their upper bounds. A Two-step All-strong-Pairs corElation queRy (TAPER) algorithm is proposed to exploit these properties in a filter-and-refine manner. Furthermore, we provide an algebraic cost model which shows that the computation savings from pruning is independent of or improves when the number of items is increased in data sets with Zipf-like or linear rank-support distributions. Experimental results from synthetic and real-world data sets exhibit similar trends and show that the TAPER algorithm can be an order of magnitude faster than brute-force alternatives. Finally, we demonstrate that the algorithmic ideas developed in the TAPER algorithm can be extended to efficiently compute negative correlation and uncentered Pearson's correlation coefficient.",

keywords = "Association analysis, Data mining, Pearson's correlation coefficient, Statistical computing",

author = "Hui Xiong and Shashi Shekhar and Tan, {Pang Ning} and Vipin Kumar",

note = "Funding Information: This work was partially supported by US National Science Foundation (NSF) grant IIS-0308264, NSF grant ITR ACI-0325949, and by the US Army High Performance Computing Research Center under the auspices of the Department of the Army, Army Research Laboratory cooperative agreement number DAAD19-01-2-0014. The content of this work does not necessarily reflect the position or policy of the government and no official endorsement should be inferred. Access to computing facilities was provided by the AHPCRC and the Minnesota Supercomputing Institute. In addition, the authors thank Professor Joydeep Ghosh (University of Texas, Austin) and Professor Ke Wang (Simon Fraser University, Canada) for valuable comments. Finally, they are grateful to Kim Koffolt for her timely and detailed feedback to help improve the readability of this paper.",

year = "2006",

month = apr,

doi = "10.1109/TKDE.2006.1599388",

language = "English (US)",

volume = "18",

pages = "493--508",

journal = "IEEE Transactions on Knowledge and Data Engineering",

issn = "1041-4347",

publisher = "IEEE Computer Society",

number = "4",

}

TY - JOUR

T1 - TAPER

T2 - A two-step approach for all-strong-pairs correlation query in large databases

AU - Xiong, Hui

AU - Shekhar, Shashi

AU - Tan, Pang Ning

AU - Kumar, Vipin

N1 - Funding Information: This work was partially supported by US National Science Foundation (NSF) grant IIS-0308264, NSF grant ITR ACI-0325949, and by the US Army High Performance Computing Research Center under the auspices of the Department of the Army, Army Research Laboratory cooperative agreement number DAAD19-01-2-0014. The content of this work does not necessarily reflect the position or policy of the government and no official endorsement should be inferred. Access to computing facilities was provided by the AHPCRC and the Minnesota Supercomputing Institute. In addition, the authors thank Professor Joydeep Ghosh (University of Texas, Austin) and Professor Ke Wang (Simon Fraser University, Canada) for valuable comments. Finally, they are grateful to Kim Koffolt for her timely and detailed feedback to help improve the readability of this paper.

PY - 2006/4

Y1 - 2006/4

N2 - Given a user-specified minimum correlation threshold θ and a market-basket database with N items and T transactions, an all-strong-pairs correlation query finds all item pairs with correlations above the threshold θ. However, when the number of items and transactions are large, the computation cost of this query can be very high. The goal of this paper is to provide computationally efficient algorithms to answer the all-strong-pairs correlation query. Indeed, we identify an upper bound of Pearson's correlation coefficient for binary variables. This upper bound is not only much cheaper to compute than Pearson's correlation coefficient, but also exhibits special monotone properties which allow pruning of many item pairs even without computing their upper bounds. A Two-step All-strong-Pairs corElation queRy (TAPER) algorithm is proposed to exploit these properties in a filter-and-refine manner. Furthermore, we provide an algebraic cost model which shows that the computation savings from pruning is independent of or improves when the number of items is increased in data sets with Zipf-like or linear rank-support distributions. Experimental results from synthetic and real-world data sets exhibit similar trends and show that the TAPER algorithm can be an order of magnitude faster than brute-force alternatives. Finally, we demonstrate that the algorithmic ideas developed in the TAPER algorithm can be extended to efficiently compute negative correlation and uncentered Pearson's correlation coefficient.

AB - Given a user-specified minimum correlation threshold θ and a market-basket database with N items and T transactions, an all-strong-pairs correlation query finds all item pairs with correlations above the threshold θ. However, when the number of items and transactions are large, the computation cost of this query can be very high. The goal of this paper is to provide computationally efficient algorithms to answer the all-strong-pairs correlation query. Indeed, we identify an upper bound of Pearson's correlation coefficient for binary variables. This upper bound is not only much cheaper to compute than Pearson's correlation coefficient, but also exhibits special monotone properties which allow pruning of many item pairs even without computing their upper bounds. A Two-step All-strong-Pairs corElation queRy (TAPER) algorithm is proposed to exploit these properties in a filter-and-refine manner. Furthermore, we provide an algebraic cost model which shows that the computation savings from pruning is independent of or improves when the number of items is increased in data sets with Zipf-like or linear rank-support distributions. Experimental results from synthetic and real-world data sets exhibit similar trends and show that the TAPER algorithm can be an order of magnitude faster than brute-force alternatives. Finally, we demonstrate that the algorithmic ideas developed in the TAPER algorithm can be extended to efficiently compute negative correlation and uncentered Pearson's correlation coefficient.

KW - Association analysis

KW - Data mining

KW - Pearson's correlation coefficient

KW - Statistical computing

UR - http://www.scopus.com/inward/record.url?scp=33644654860&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33644654860&partnerID=8YFLogxK

U2 - 10.1109/TKDE.2006.1599388

DO - 10.1109/TKDE.2006.1599388

M3 - Article

AN - SCOPUS:33644654860

SN - 1041-4347

VL - 18

SP - 493

EP - 508

JO - IEEE Transactions on Knowledge and Data Engineering

JF - IEEE Transactions on Knowledge and Data Engineering

IS - 4

ER -

TAPER: A two-step approach for all-strong-pairs correlation query in large databases

Abstract

Bibliographical note

Keywords

Access

OpenUrl availability

Other files and links

Fingerprint

Cite this