For all the grief Miller got, he was pretty good at getting lefties out.
wOBA v. LHB: .279
wOBA v. RHB: .356
Remember, that's on an OBP scale. To put this into perspective, his performance against left-handers is nearly equal to Kenji Johjima's 2008 season, but against righties he was facing Russell Martin. Unfortunately for Miller, his PAs were only off-set by 20. Of course, most of his RH at-bats came in mop-up time, but his overall stats took a beating because of it.
Another reason for his down coming was his salary and type-B status. If the Rays offer him ARB and he declines the team stands to gain a supplemental pick.
Unless of course he accepts and feels he can't earn more on the open market, to which the Rays may or may not get Miller for 2 mil. Perhaps there's a wink-nod agreement, but the downside to Miller accepting is taking up a larger chunk of salary.