Defence barrister Simon Ray, who stepped in to give the closing speech because Harris's QC Sonia Woodley is ill, also told jurors Harris was being subjected to a prosecution that fails to meet the standard of criminal proof.
"Making allegations loudly and forcefully does not make them true," he told Southwark Crown Court on Monday.
"You may think that a prosecutor who was more confident of the evidence would not have had to resort to name-calling of the defendant.
"The reality is that when you take a cold hard look at the prosecution case, there are problems that cannot be pushed aside."
He said those problems created "unavoidable doubt" over the truth of the claims.
"The prosecution has to prove it and they have to prove it so that you are sure, and that's a high standard. A standard that when you analyse the evidence, which is all that any of us can do, the prosecution has not met."
Harris is accused of 12 counts of indecent assault between 1968 and 1986, all of which he denies.
Mr Ray said the entertainer was at "a serious disadvantage" due to the time lapse since the claims.
"The passage of time, which is a gap of up to 45 years ... is a serious disadvantage to the defendant, and that would be true whether he was rich or poor, famous or unknown."
Mr Ray told the jury of six men and six women that Harris faced an impossible situation trying to remember details from so long ago.
"Mr Harris can't really win. If he fails to remember things, any incorrect assertions by him are damned as deliberate lies. If he does remember details and tries to say so, again that's dismissed."
During the trial, the court has heard details of two extra-marital affairs Harris was involved in, including one with an alleged victim.
Mr Ray said: "He has been punished for his infidelity by, effectively, public humiliation."
Jurors have been told by prosecutor Sasha Wass QC that Harris had "a dark side" to his personality.
Mr Ray said: "Everyone has a private life, everyone has a side to their personality that they only reveal to those that know them intimately, and, to be honest, maybe not anyone."
The first count on the indictment concerns an allegation by a woman who claims Harris groped her at an event near Portsmouth between 1968 and 1970.
Mr Ray said: "The reality is that there is no evidence from any independent source that the event that (the alleged victim) described took place."
The alleged victim in the second count claims that Harris touched her bottom when she was waitressing at a celebrity event in Cambridge in 1975 - although it was later suggested that this could have occurred in 1978.
Mr Ray told the jury: "Was she 13 or was she 16? Those three years for a teenager are a real and significant difference and not one that you would get wrong."
Harris told the jury that he had not visited the city until 2010, but was later forced to admit that he had appeared in an episode of Star Games filmed there in 1978.
This was branded by the prosecution a deliberate lie, but Mr Ray said Harris would never have been foolish enough to lie if he had remembered the show.
"No defendant would be that stupid," jurors were told.
"Precisely because that opens him up to the sort of criticisms that he has had to face."