You know how most VN's don't voice the main char? It's left up to you to decide what he sounds like?
I don't watch animes based off VN, cause it'll give me a bad impression, but from the previews I've seen, they're voiced in it.
I don't know if it's like when a book is translated into a movie and people don't like the voice a guy has in it, but what if, instead of voicing him, they just made him move lips?
Maybe just subtitle what he's saying in, or have the characters look at him questioningly, maybe break the 4th wall, I'm not sure, but somehow reference the VN. I just thought how funny it'd be to see.
He'd be moving his lips, giving his whole speech, and maybe people would just stare? Or you'd hear someone say something about "I never hear xxx speaking..."
Not sure why I came up with this, could be from watching Suzumiya, or something, but I just generally thought that if the movie principle applies that you don't like the guy who voices a character, maybe it could apply in this situation?