Pose has been shown to be a discriminative cue for inferring activities. The project aims at improving action recognition using pose as input. Instead of relying on 3D skeletons from a depth camera, our approach takes in heatmaps from an off-the-shelf pose extractor enabling application on standard action datasets. Our current model improves over the state of the art pose based models on JHMDB, HMDB and Charades through enhanced temporal features, effective data augmentations and improved modeling of joint motion information by focusing on most discriminative joints.