Sri Lanka beat the Black Caps by 63 runs in the first test to take a 1-0 lead in the two-match series in Galle.
New Zealand lasted just four overs on day five before being dismissed for 211 in pursuit of 275 for victory.
With only two wickets in hand and needing 68 more runs to win, the Black Caps had their hopes pinned on Rachin Ravindra who started the final day unbeaten on 91.
Ravindra could only add one run to his overnight score before spinner Prabath Jayasuriya had him out LBW.
Number 11 batter Will O’Rourke lasted six balls before Jayasuriya bowled him for a duck.
Jayasuriya (5-68) and Ramesh Mendis (3-83) shone with the ball for the hosts.
The second test, also in Galle, begins on Thursday.