Coventry should have taken the lead after 15 minutes when Gary McSheffrey ran on to a Steve Staunton through-ball, but was tripped from behind in the area by Robbie Stockdale.
McSheffrey picked himself up to take the spot-kick but his low effort to Mike Pollitt's left was comfortably saved by the Millers keeper.
Just before the half-hour mark, it was Rotherham's turn to come close to taking the lead.
Coventry keeper Scott Shearer made a hash of reaching for a cross and Shaun Barker headed into an empty net, but referee Chris Foy ruled the effort out for an apparent foul on the keeper.
They came close minutes later when two Jody Morris free-kicks were met well by Chris Swailes, whose header rattled the upright, then by Barker, but his effort flew narrowly wide.
As Rotherham looked to take a lead into half- time, Michael Proctor spotted Shearer off his line, but his fine lob from 25 yards landed on the roof of the net.
Half-time saw the introduction of Andy Morrell for Julian Joachim as struggling City looked to find some cohesion.
Coventry did look slightly more motivated after the break and Graham Barrett came close on 50 minutes after he shot from 18 yards, but it was high and wide.
Morrell then had a mis-hit effort cleared off the line by Stockdale before he grabbed the lead on 55 minutes.
Barrett charged down the left wing and pulled the ball across for Morrell who steered it past Pollitt from close range.
However the lead was short lived and Chris Sedgwick, a 72nd-minute substitute, made an instant impact when he turned Morris' free-kick back across the area for Martin Butler to head in from close range.