I was thinking, would it be possible to modify Joomla sufficiently. I haven't worked with it before and i don't want to end up with a square peg situation.
The other option being using frameworks Django perhaps to build it from scratch.
What would you people suggest?
Build from the database out. And please use encryption appropriately.
In the US at least you are dealing with HIPAA which has a deep set of requirements your system will have to meet.
And you are dealing with crucial information, if you screw up medical records such that crucial information is left out or garbled or lost you could potentially kill people.
So use tools that give you a clear view into the development process and let you track, record and correct the system as you are building it.
Use whatever language the team is most familiar with and don't think that any framework will make your work easier unless you already know that framework well.