Having taught basic `aqidah courses several times over, through phone, internet, and the time honored medium of person to person contact, I can tell you that one of the trickiest subjects to tackle is always that of qadr or divine predestination: are we in control of our own lives, or are we living out a hard deterministic cosmic script in which we are bound without any choice of our own?
The brief answer according to orthodox Sunni theology based on the Qur'an and Sunnah is:
(a) Allah knew everything before He created anything, in both gross and minute detail, that which was, that which is, and that which shall for ever be through His perfect and all-encompassing knowledge, which is an essential attribute of His divine essence.
(b) Everything that ever happened only happened, happens and shall happen through His causing it to happen. He has control over what happens in terms of times, outcomes, measurements, amounts, quanta, etc., to the minutest degree. Nothing happens in creation against His will or according to anything other than His plan. We do not believe like the misguided nations before us, or their heretical conjugate counterparts within our fold, that anything happens against His will, or that he regrets or resents anything that happens. Indeed the meaning of the divine name al-Samad mentioned in surat al-Ikhlas is He through whom if something is to happen, it must be done, and He, who if He doesn't partake in a matter, that matter will never be realized.
(c) Every human being has the ability to chose between good and evil at every turn. Those choices earn a person a place in the hereafter which is commensurate with the quality of said choices made.
Those choices plays out in one of five combination scenarios.
1- Tawfiq: This is when you wish for good, and Allah wishes for you to execute that good you wish for. This results in earning a minimum of ten times that good deed's value in reward, up to 700 and beyond, as well as having further doors for good opened.
2- Reward without tawfiq: This is when you wish for good, but Allah doesn't allow you to execute what you wished for. This results in earning the good deed intended at face value.
3- Null result: This is when someone wishes for something which is morally neutral like buying lavender soap rather than pine. The result is likewise null whether the person gets what they wanted or not.
4- Protection from evil: This is when one wishes to sin and indeed goes through with the attempt, but is thwarted by Allah's not wishing for that sin to be actualized. From Allah's grace, such a person is not punished for such a matter.
5- Khudhlan: This is when a person wishes to sin, and Allah allows them to carry out such a self-destructing act so that they can be indisputably deserving of the punishment that they earn by their own hands on the day of Judgment. Such an act is punished at face value.
Now when taken in isolation, each (a), (b), and (c), seem very straightforward and even intuitive. The problem lies in reconciling, how they can all be true at the same time.
The messenger of Allah, sallallahu`alayhi wa sallam, informed us that the reconciliation would not be graspable by the mind which is trapped in the cage of time and space, so we should save our time and bandwidth from being sunk in this question. There is great wisdom in his prescription: it allows one, by accepting their inability to grasp the common thread that binds three such basic and intuitive concepts, to come to terms with the greater and more important realization that Allah has the prerogative to chose what He chooses without having to answer to others: "[None have such right that] He be asked about what He does, but [He has the right that] they be asked." (surat al-Anbiya').
Far from being a crippling blow to the intellect, the acceptance of (a), (b), and (c) being true at the same time is as much a sign of suspending one's intellect as using a cell phone while not being able to explain how it works in complete detail. To conceive of the idea that some parts of the mechanics of creation might be beyond the grasp of most, if not all of the creation is a given when considering that in physical scale, the laws that govern the seven firmaments, heaven, hell, multiple dimensions (which are talked about in the hadith literature), is not far-fetched. The laws of predestination apply to all of them universally, and are therefore functioning at a fundamental level that to which we have not been, and, in this life, are not to be exposed.
This is in line with a saying attributed to Sayyiduna al-Hasan bin Ali bin Abi Talib by Shaykh Ali Hajwiri in his Kashf al-Mahjub: "I saw the position of the companions of the messenger of Allah sallallahu`alayhi wa sallam with regards to predestination was to believe in their hearts that Allah was in total control of all affairs, while practically conducting themselves as if they were in complete control of their affairs."
The messenger of Allah, sallallahu`alayhi wa sallam said, "Each is facilitated to [behave in the manner for] which he was created." This is with regards to the idea that in Allah's predestination, each soul was divinely preordained to either be from the people of felicity or wretchedness. With regards to the issue of fate, I have found that there are two reactions in people, each being indicative of one of the two aforementioned states. Those with propensity towards wretchedness will respond by saying that if they are going to hell anyways, then there is no point in even trying anymore. Those who have a propensity towards felicity will understand that the only One who has the power to switch their fate one way or the other is Allah, and so this realization will increase them in their humility toward Him, by highlighting the knowledge that He is the only one who can help them out in this matter.
To question whether such a system is fair, or why it is in place is to run afoul of the Qur'anic maxim of surat al-Anbiya' mentioned earlier: the Muslim theological definition of justice is that to which Allah commands. The fact that that command is one in which there is great mercy and benefit to us is not an essential quality of Justice, but an incidental one which Allah chose through His generosity. We do not, like the neo-Platonic mu`tazilah or the Christians, hold Allah to an external and synthetic standard of justice. If He was beholden to anything, it, rather than He would be the supreme power in the universe and more worthy of worship and veneration, which, as a concept is alien to the prophetic tradition and will lead an utterly ludicrous and paradoxical trap regarding the nature of divinity.
Finally coming to the question indicated in the title, can supplication or du`a change one's fate, the authentic hadith literature would seem to indicate so, as Hakim narrates a hadith which Dhahabi accepts as sahih: "On the authority of Thawban, [may Allah be pleased with him], the messenger of Allah, sallallahu`alayhi wa sallam said, 'Nothing can repel fate, except du`a, and nothing can increase one's lifespan, except piety. Verily a man will be deprived of his apportioned lot in this world due to sins he commits.'"
Does this mean that du`a is a wildcard in the system? Hardly. Rather the meaning is there to reaffirm (c) which is the idea that the choices that you and I make are very intimately tied to the outcomes that we experience in this world and the hereafter, by stating the benefits and power of an act of piety such as du`a or the harms of an impiety like sin. Does this necessarily negate (a) or (b)? Not at all. Allah knows that the person making du`a will make du`a and He allows it to happen. The fate that is repelled when such a person makes du`a is that evil fate that they would have had had (a), (b), and (c) not lined up in such a way to save him from it by means of du`a.
Perhaps this is what is meant in the hadith of Bukhari narrated by Sayyiduna Anas, may Allah be pleased with him: "The Messenger of Allah, sallallahu`alayhi wa sallam, said, 'Indeed the slave, when placed in his grave, and his companions turn around and leave him and he hears the footfalls of their sandals, at that time two angels will come to him, sit him up, and ask him, 'What did you used to say about this man, Muhammad, sallallahu`alahyhi wa sallam?' As for the believer, he will say, 'I testify that he is the slave of Allah and His messenger.' It will be said to him, 'Look at your your seat in the fire. Indeed Allah has exchanged for it a place in jannah.' He will be able to see both of them at the same time..." One possible fate was replaced by another through piety, all the while with Allah's primordial knowledge of one taking precedence over the other, and His allowing and facilitating one to overtake the other.
... and Allah knows best.