Berwick missed a chance to go above idle Cowdenbeath as they couldn't put the ball in the net despite numerous chances.
John Coughlan's men hammered the Fifers 4-0 on the first day of the season and forced the home side back inside two minutes.
John Dodds fumbled Mark McGarty's free kick and didn't look too clever with a low drive from Ian little shortly after.
East Fife missed a great opportunity when John Martin raced away on the right but he delayed his shot enough for Grant McNicoll to make the tackle.
The wee Rangers knocked the ball about smartly and on 27 minutes Kevin Gordon picked out Chris McGroaty but he miscued his volley well wide.
Gordon then threatened with a brave diving header and Little rattled a shot which climbed just over the bar after the break.
It was all Berwick and McGroaty had a low drive tipped round the post and Hutchison screwed a shot wide after the ball broke in the box.
Greig McDonald swivelled in the box and tested the keeper with eight minutes left as the Fifers finally got a shot on target.
They stole all the points when sub Paliczka cut in from the left and smacked a low shot past the keeper.
In the very last seconds John Martin lashed out at a Rangers player and was red carded.