SAN FRANCISCO, May 21— Eric Hoffer, an itinerant farm worker who became a longshoreman and whose musing on life, power and social order turned him into a celebrated author-philosopher, died today at his home here. He was 80 years old.
Years ago, after he had become famous and the subject of interviews, he roared back his answer to a question that began with the premise ''you're an intellectual,'' saying, ''No! I'm a longshoreman.''
Mr. Hoffer's first book, ''The True Believer,'' was published in 1951. He continued working along the docks as a member of the International Longshoreman's and Warehouseman's Union and wrote between work assignments.
He left the docks in 1967. In 1970 he withdrew from public life, saying: ''I'm going to crawl back into my hole where I started. I don't want to be a public person or anybody's spokesman. I am not the type for it and I dislike it.'' Getting Off the Train
His withdrawal ended his lectures at the University of California at Berkeley, as well as his column in The San Francisco Examiner and his appearances on KPTV, the public television station here.
He said of his decision to withdraw, ''Any man can ride a train. Only a wise man knows when to get off.'' About that time he said in an interview with Norman Wilner of the North American Newspaper Alliance, as he discussed the intellectuals among his recent contacts: ''I think all these college boys and college professors have been brainwashed. Plato, they say, was a great genius, but Plato only dealt in half-truths, he shillied and he shallied and he didn't accomplish a thing. He was an intellectual.''
Asked to comment on Freud, he shrugged and said: ''Ah, don't talk to me about Freud. Freud lived in a tight little circle in Vienna and inside that tight little circle was another tight little circle and inside that tight little circle was still another tight little circle. What applies to that poor man, Freud, does not necessarily apply to me.'' Invitation From Johnson
In October 1967 he was given a tour of the White House by President Johnson who had invited him after seeing the longshoreman on a television program.
Mr. Hoffer told Mr. Johnson not to pay heed to the polls that showed his policies losing public support. He also told reporters that he had never spent a day in a formal school. The next year Mr. Johnson put Mr. Hoffer on a Presidential commission to explore violence and protest. As a member of that panel Mr. Hoffer walked out of a hearing when a student from Berkeley urged the study of the causes of protest, the Vietnam War. Mr. Hoffer's attacks on antiwar protesters made him a target of members of the liberal left.
When President Reagan honored Mr. Hoffer last February by awarding him the Presidential Medal of Freedom, the nation's highest civilian award, he recalled a visit from Mr. Hoffer when he was Governor of California. ''I got some pretty good, sound and salty advice,'' Mr. Reagan said.
After becoming established by the reception to his first book, Mr. Hoffer continued to write to the end of his life. His published works included ''The Passionate State of Mind'' (1955); ''The Ordeal of Change'' (1963); ''Temper of Our Time'' (1967); ''Working and Thinking on the Waterfront'' (1967); ''First Things, Last Things'' (1970); ''Reflections on the Human Condition'' (1972); ''In Our Time'' (1976), and ''Before the Sabbath'' (1979). Partly Blind as Child
Mr. Hoffer was born in New York City July 25, 1902. He often attributed his love of reading to the fact that he was partly blind as a child as a result of a fall. His sight mysteriously returned at the age of 15. He spent 23 years as a migratory worker on farms, in mines and in odd jobs. In World War II, in 1943, he obtained membership in the longshoremen's union where he remained as a dock worker until he was forced into retirement at the age of 65.
Much of his writing was in epigrams, such as this on Communists: ''He who clings with all his might to an absolute truth fears compromise more than the devil.'' Or this: ''Communism is capitalism gone berserk.''
And on himself: ''The main thing is not to take myself seriously.''
Illustrations: photo of Eric Hoffer