I mean, I've been in peoples' houses before where I was left alone (or just wanted the luxury of losing myself in things rather than the people I was surrounded with), where there were books that spoke to my geekish interests. And even I would have had to know the person before I felt comfortable just picking those volumes up. For Mary to pick up any book at all is personal, invasive, especially for a famiily she doesn't know that well at all. They're not even her in-laws, and even John has barely met them before that Christmas. But for her to pick up this particular book strikes me as very odd indeed, because it's not going to be at all accessible and interesting. Except, possibly, to another mathematician.
Ergo, Moriarty. From "The Final Problem": At the age of twenty-one [Moriarty] wrote a treatise upon the Binomial Theorem, which has had a European vogue. On the strength of it he won the Mathematical Chair at one of our smaller universities, and had, to all appearance, a most brilliant career before him. But the man had hereditary tendencies of the most diabolical kind. A criminal strain ran in his blood, which instead of being modified, was increased and rendered infinitely more dangerous by his extraordinary mental powers."
Mary's character on the BBC show has always struck me as very smart, very clever. She's the one who identifies the cypher in the James-or-John message in TEH, she figures out Sholto's room in TSOT, she gets past Magnussen's security. And she re-invents herself at least once, probably multiple times if the glimpses we get in Moriarty's mind palace are any indication. She either got past Mycroft for several months until the wedding (at a minimum), or somehow had cornered him well enough that he didn't tell Sherlock anything about her. This takes cleverness, but it's cleverness that's just making her criminality that much more effective. It could be that Mary is so used to manipulating her surroundings that she sees something with her host's name on it and jumps on it.
On the other hand, this is their house. She's surrounded by photos of family vacations, kids in fuzzy jumpers, and knick-knacks from a family's lifetimes, and she picks up on a specialist book written by a mathematics professor? That's... odd, to say the least.
If I was inclined to go all that far down this road (and I'm not, really), I'd think back on what Sherlock said to the cabbie back in ASIP: asking whether Moriarty was a man, an organization, what? we all assume Moriarty is Andrew Scott's character because he fooled pretty much everyone into thinking that was a lie and then was unmasked. It's entirely consistent with what the show's actually established, though, for Moriarty not to be just one individual, and for Mary to be part of it.
File this into the "too clever by half for Moffat to actually pull off, so please let this not be the direction he's heading, but still a lot of fun to play with the idea" category.