A 90th-minute winner by Mason Walsh earned Bognor their first away league win of the season at the home of leaders Tonbridge.
Pompey loanee Brad Lethbridge put the Rocks ahead early in the second half but Alex Read levelled it up midway through the half.
It looked like the Rocks sixth successive away draw was on the cards but Walsh pounced in the dying moments to make it five wins and five draws for Bognor, who are now third in the table and just two points behind the team they beat.
The Rocks now have two home games in a week - they host Folkestone on Tuesday and Margate next Saturday.
A couple of our local sides were in FA Vase action - Pagham won 4-2 at Hanworth Villa thanks to goals from Andy Chick, Callum Overton (2) and George Gaskin.
Sidlesham put up a brave battle but went down 2-0 at home to Chatham in the same competition.
In the SCFL premier Chichester City lost 2-1 at home to Eastbourne Town, with only a late Connor Cody goal to show for their efforts.
In division one Selsey hammered Billingshurst 6-0 at the High Street Ground, with Ryan Morey (2), Dan Bassil and Tom Jefkins (2) on target after an own goal had set the ball rolling.
Bosham crashed to a 7-0 defeat at Rustington in the Sussex Intermediate Cup.
Get all the local football on this website throughout the week and in the Observer every Thursday