] or is it just in the movie that Marianne asks why W. will not shake hands with her. I seem to remember that it's in the novel too.

Indeed it is. In S&S it's presumably a gesture between people who are on close terms. The Tarleton quote seems to show that handshaking was usual between people who were not so intimate as Marianne and Willoughby (I presume this because Tarleton wouldn't have needed to reassure his intimate friends).