You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.
A high performance implementation of HDBSCAN clustering.
Notifications You must be signed in to change notification settings
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Go to fileHDBSCAN - Hierarchical Density-Based Spatial Clustering of Applications with Noise. Performs DBSCAN over varying epsilon values and integrates the result to find a clustering that gives the best stability over epsilon. This allows HDBSCAN to find clusters of varying densities (unlike DBSCAN), and be more robust to parameter selection. In practice this means that HDBSCAN returns a good clustering straight away with little or no parameter tuning -- and the primary parameter, minimum cluster size, is intuitive and easy to select. HDBSCAN is ideal for exploratory data analysis; it's a fast and robust algorithm that you can trust to return meaningful clusters (if there are any). Based on the papers:
McInnes L, Healy J. Accelerated Hierarchical Density Based Clustering In: 2017 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, pp 33-42. 2017 [pdf] R. Campello, D. Moulavi, and J. Sander, Density-Based Clustering Based on Hierarchical Density Estimates In: Advances in Knowledge Discovery and Data Mining, Springer, pp 160-172. 2013
The hdbscan package inherits from sklearn classes, and thus drops in neatly next to other sklearn clusterers with an identical calling API. Similarly it supports input in a variety of formats: an array (or pandas dataframe, or sparse matrix) of shape (num_samples x num_features) ; an array (or sparse matrix) giving a distance matrix between samples.
import hdbscan from sklearn.datasets import make_blobs data, _ = make_blobs(1000) clusterer = hdbscan.HDBSCAN(min_cluster_size=10) cluster_labels = clusterer.fit_predict(data)
Significant effort has been put into making the hdbscan implementation as fast as possible. It is orders of magnitude faster than the reference implementation in Java, and is currently faster than highly optimized single linkage implementations in C and C++. version 0.7 performance can be seen in this notebook . In particular performance on low dimensional data is better than sklearn's DBSCAN , and via support for caching with joblib, re-clustering with different parameters can be almost free.
All of which come equipped with methods for plotting and converting to Pandas or NetworkX for further analysis. See the notebook on how HDBSCAN works for examples and further details.
The clusterer objects also have an attribute providing cluster membership strengths, resulting in optional soft clustering (and no further compute expense). Finally each cluster also receives a persistence score giving the stability of the cluster over the range of distance scales present in the data. This provides a measure of the relative strength of clusters.
The HDBSCAN clusterer objects also support the GLOSH outlier detection algorithm. After fitting the clusterer to data the outlier scores can be accessed via the outlier_scores_ attribute. The result is a vector of score values, one for each data point that was fit. Higher scores represent more outlier like objects. Selecting outliers via upper quantiles is often a good approach.
Based on the paper: R.J.G.B. Campello, D. Moulavi, A. Zimek and J. Sander Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection, ACM Trans. on Knowledge Discovery from Data, Vol 10, 1 (July 2015), 1-51.
The hdbscan package also provides support for the robust single linkage clustering algorithm of Chaudhuri and Dasgupta. As with the HDBSCAN implementation this is a high performance version of the algorithm outperforming scipy's standard single linkage implementation. The robust single linkage hierarchy is available as an attribute of the robust single linkage clusterer, again with the ability to plot or export the hierarchy, and to extract flat clusterings at a given cut level and gamma value.
import hdbscan from sklearn.datasets import make_blobs data, _ = make_blobs(1000) clusterer = hdbscan.RobustSingleLinkage(cut=0.125, k=7) cluster_labels = clusterer.fit_predict(data) hierarchy = clusterer.cluster_hierarchy_ alt_labels = hierarchy.get_clusters(0.100, 5) hierarchy.plot()
Based on the paper: K. Chaudhuri and S. Dasgupta. "Rates of convergence for the cluster tree." In Advances in Neural Information Processing Systems, 2010.
The hdbscan package supports a branch-detection post-processing step by Bot et al.. Cluster shapes, such as branching structures, can reveal interesting patterns that are not expressed in density-based cluster hierarchies. The BranchDetector class mimics the HDBSCAN API and can be used to detect branching hierarchies in clusters. It provides condensed branch hierarchies, branch persistences, and branch memberships and supports joblib's caching functionality. A notebook demonstrating the BranchDetector is available.
import hdbscan from sklearn.datasets import make_blobs data, _ = make_blobs(1000) clusterer = hdbscan.HDBSCAN(branch_detection_data=True).fit(data) branch_detector = hdbscan.BranchDetector().fit(clusterer) branch_detector.cluster_approximation_graph_.plot(edge_width=0.1)
Based on the paper: D. M. Bot, J. Peeters, J. Liesenborgs and J. Aerts "FLASC: A Flare-Sensitive Clustering Algorithm: Extending HDBSCAN* for Detecting Branches in Clusters" Arxiv 2311.15887, 2023.
Easiest install, if you have Anaconda (thanks to conda-forge which is awesome!):
conda install -c conda-forge hdbscan
PyPI install, presuming you have an up to date pip:
pip install hdbscan
Binary wheels for a number of platforms are available thanks to the work of Ryan Helinski .
If pip is having difficulties pulling the dependencies then we'd suggest to first upgrade pip to at least version 10 and try again:
pip install --upgrade pip pip install hdbscan
Otherwise install the dependencies manually using anaconda followed by pulling hdbscan from pip:
conda install cython conda install numpy scipy conda install scikit-learn pip install hdbscan
For a manual install of the latest code directly from GitHub:
pip install --upgrade git+https://github.com/scikit-learn-contrib/hdbscan.git#egg=hdbscan
Alternatively download the package, install requirements, and manually run the installer:
wget https://github.com/scikit-learn-contrib/hdbscan/archive/master.zip unzip master.zip rm master.zip cd hdbscan-master pip install -r requirements.txt python setup.py install
The package tests can be run after installation using the command:
nosetests -s hdbscan
or, if nose is installed but nosetests is not in your PATH variable:
python -m nose -s hdbscan
If one or more of the tests fail, please report a bug at https://github.com/scikit-learn-contrib/hdbscan/issues/new
The hdbscan library supports both Python 2 and Python 3. However we recommend Python 3 as the better option if it is available to you.
For simple issues you can consult the FAQ in the documentation. If your issue is not suitably resolved there, please check the issues on github. Finally, if no solution is available there feel free to open an issue ; the authors will attempt to respond in a reasonably timely fashion.
We welcome contributions in any form! Assistance with documentation, particularly expanding tutorials, is always welcome. To contribute please fork the project make your changes and submit a pull request. We will do our best to work through any issues with you and get your code merged into the main branch.
If you have used this codebase in a scientific publication and wish to cite it, please use the Journal of Open Source Software article.
L. McInnes, J. Healy, S. Astels, hdbscan: Hierarchical density based clustering In: Journal of Open Source Software, The Open Journal, volume 2, number 11. 2017
@articlemcinnes2017hdbscan, title=hdbscan: Hierarchical density based clustering>, author=McInnes, Leland and Healy, John and Astels, Steve>, journal=The Journal of Open Source Software>, volume=2>, number=11>, pages=205>, year=2017> >
To reference the high performance algorithm developed in this library please cite our paper in ICDMW 2017 proceedings.
McInnes L, Healy J. Accelerated Hierarchical Density Based Clustering In: 2017 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, pp 33-42. 2017
@inproceedingsmcinnes2017accelerated, title=Accelerated Hierarchical Density Based Clustering>, author=McInnes, Leland and Healy, John>, booktitle=Data Mining Workshops (ICDMW), 2017 IEEE International Conference on>, pages=33--42>, year=2017>, organization=IEEE> >
If you used the branch-detection functionality in this codebase in a scientific publication and which to cite it, please use the Arxiv preprint:
D. M. Bot, J. Peeters, J. Liesenborgs and J. Aerts "FLASC: A Flare-Sensitive Clustering Algorithm: Extending HDBSCAN* for Detecting Branches in Clusters" Arxiv 2311.15887, 2023.
@miscbot2023flasc, title=FLASC: A Flare-Sensitive Clustering Algorithm: Extending HDBSCAN* for Detecting Branches in Clusters>, author=D. M. Bot and J. Peeters and J. Liesenborgs and J. Aerts>, year=2023>, eprint=2311.15887>, archivePrefix=arXiv>, primaryClass=cs.LG>, url=https://arxiv.org/abs/2311.15887>, >
The hdbscan package is 3-clause BSD licensed. Enjoy.