A fine-grained scheduling strategy for improving the performance of parallel frequent itemsets mining

We propose a scheduling strategy in this paper to address the load imbalance problem of the distributed parallel apriori (DPA) algorithm published recently. We use fine grained tasks that are derived by dividing the tasks defined by DPA into smaller subtasks. The subtasks will be scheduled by a dynamic self-scheduling scheme for better load balance. Furthermore, we propose two different methods for data transmission from the master to workers. The first one broadcasts all the frequent k-itemsets to all work nodes while the second one transmits only the required data to each individual work node. Experimental results demonstrate the proposed two approaches both outperform DPA. The first one is more suitable for small datasets and the second one provides steadier performance improvement no matter which self-scheduling scheme is adopted.