Sleep-wake classification is important for measuring the sleep quality. In this paper, we propose a novel deep learning framework for sleep-wake detection by using acceleration and heart rate variability (HRV) data. Firstly, considering the high sampling rate of acceleration data with temporal dependency, we propose a local feature based long short-term memory (LF-LSTM) approach to learn high-level features. Meanwhile, we manually extract representative features from HRV data, as HRV data has a distinct format with acceleration data. Then, a unified framework is developed to combine the features learned by the LF-LSTM from acceleration data and the features extracted from HRV data for sleep-wake detection. We use real data to evaluate the performance of the proposed framework and compare it with some benchmark approaches. The results show that the proposed approach achieves a superior performance over all the benchmark approaches for sleep-wake detection.