Chris Columbus, who directed the previous films, was offered also to direct this one, but he declined in order to spend more time with his children. He did accept to produce the film.
Kenneth Branagh, who had previously played Professor Lockhart, was offered the director's chair. Had he accepted, he would've directed his ex-wife Emma Thompson.
Guillermo del Toro was offered the film, but he declined. Del Toro actually wanted to direct, but felt that the movie's 'verse didn't conform to his more "Dickensian" vision of the books. He opted to work on Hellboy (2004) instead, but recommended his friend Alfonso Cuarón, and the rest is history. Ironically, Cuarón initially refused, complaining that he had not even read the books or watched the previous films, so Del Toro told him to do so then and shut up, and Cuarón eventually found out he loved the story.
Marc Forster of Finding Neverland fame was offered to direct, but he declined because he didn't want to go through the experience of working with children again.
Tilda Swinton was approached for the part of Professor Trelawney, but turned it down. At first, it was reported she did it due to scheduling conflicts, but she later revealed not to be a fan of Harry Potter anyway. Swinton has criticized the saga for romanticizing boarding schools, a concept she hates due to her own experiences with it.
After Richard Harris died, and also earlier while his health was declining, Ian McKellen was approached to play Dumbledore, but he turned it down, as Harris was famously critical of McKellen's acting (calling him "dreadful" and "passionless") and McKellen felt it would have been disrespectful to inherit his role. He has always maintained that "Richard Harris was mainly a disappointed man because I had played Gandalf and he had to settle for Dumblewit. Or Dumblebore, I should say." McKellen, who coincidentally was reading Harry Potter at the time of his work in The Lord of the Rings, further said, when asked who would win in a fight between the two, "Gandalf, of course."
Harris' family wanted Peter O'Toole to take up the role, as he was one of Harris' best friends. The studio decided against it, fearing the similarly elderly and ill actor might also die before the series was out. Ironically, he didn't - he instead passed away in 2013, almost 2 and a half years after the final film premiered.
Christopher Lee was also offered to play Dumbledore, but he refused due to all the drama and gossip that had been created around the recast, which he believed would be disrespectful to Harris.
One of Alfonso Cuarón's more bizarre ideas involved a scene with a pipe organ being played by tiny people jumping on the keyboard. Rowling rejected it, saying it wouldn't make sense in her universe.
Originally, the Trio was supposed to watch the execution of Buckbeak from a graveyard. Rowling rejected this because she envisioned the Hogwarts graveyard being in a different part of the grounds and said it would become significant in the sixth book. Therefore, the graveyard got changed to the giant sundial you see in the final film.
The original script contained mention as to why Harry's patronus is a stag.
The original script called for Hermione to slap Draco as she does in the book. During a rehearsal, Emma Watson actually slapped Tom Felton with full force. She later said she had no idea why she did it, and felt horrible about it afterwards. This is why she punches him instead in the final version (as a punch is easier to fake than a slap).
Alfonso Cuarón wanted the Dementors to be achieved through puppetry. Once it became clear that wouldn't be feasible, he switched to CGI. However, they did use footage of puppets underwater for reference when creating the CGI.
Although Azkaban doesn't actually appear in the story, concept art of it was created - as a giant stone fortress on the edge of a large waterfall. This design was later used in Harry Potter and the Order of the Phoenix when the prison actually appears - though situated on an island in the sea, as in the book.
Lucius Malfoy would have appeared for Buckbeak's execution scene, but Jason Isaacs was unavailable.
but felt that the movie's 'verse didn't conform to his more "Dickensian" vision of the books. He opted to work on Hellboy (2004) instead, but recommended his friend Alfonso Cuarón, and the rest is history. Ironically, Cuarón initially refused, complaining that he had not even read the books or watched the previous films, so Del Toro told him to do so then and shut up, and Cuarón eventually found out he loved the story.