It turned out a very silly behavior of Visual Studio 2015. Whenever you build/rebuild your solution it doesn't update WSP file. However if you want to redeploy your solution then Visual Studio's "Deploy" option force an update on WSP file. Since I am performing an upgrade on my features "Re - Deploy" was not an option.
After finding this behavior initially I thought of using Ms-build to create a wsp file and get the work done and this is what people normally do. But somewhere in my mind I was puzzled what if I have to use Visual Studio for this? Turn out to be very simple option. You can use "Publish" your solution to file system. It forces a new build and proper update of artifacts within WSP.
