A study, in the journal Sleep, said most children had nightmares, but persistent ones may be a sign of something more serious.
Having night terrors – screaming and thrashing limbs while asleep – also heightened the risk.
The charity YoungMinds said it was an important study which may help people detect early signs of mental illness.
Nearly 6,800 people were followed up to the age of 12.
Parents were regularly asked about any sleep problems in their children and at the end of the study the children were assessed for psychotic experiences such as hallucinations, delusions and thinking their thoughts were being controlled.
The study showed that the majority of children had nightmares at some point, but in 37% of cases, parents reported problems with nightmares for several years in succession.
One in 10 of the children had night terrors, generally between the ages of three and seven.
The team at the University of Warwick said a long-term problem with nightmares and terrors was linked to a higher risk of mental health problems later.
Around 47 in every 1,000 children has some form of psychotic experience.
However, those having nightmares aged 12 were three-and-a-half times more likely to have problems and the risk was nearly doubled by regular night terrors.
One of the researchers, Prof Dieter Wolke, told the BBC: “Nightmares are relatively common, as are night terrors, it is quite normal, but if they persist then there maybe something more serious about it.”
The relationship between sleep problems and psychosis is not clear.
One theory is that bullying or other traumatic events early in life can cause both symptoms.
Or the way some children’s brains are wired means the boundaries between what is real and unreal, or sleeping and wakefulness, are less distinct.
It means treating the sleep issues may not prevent psychotic events.
However, nightmares may act as an early warning sign of future, more serious, problems.
Prof Wolke said a regular routine and quality sleep were key to tackling nightmares: “Sleep hygiene is very important, they should have more regular sleep, avoid anxiety-promoting films before bed and not have a computer at night.”
Night terrors occur at specific points during sleep and can be managed by briefly waking the child.
Lucie Russell, the director of campaigns at YoungMinds, said: “This is a very important study because anything that we can do to promote early identification of signs of mental illness is vital to help the thousands of children that suffer.
“Early intervention is crucial to help avoid children suffering entrenched mental illness when they reach adulthood.”