Danny Lyon's inside shots

New York, 1980: ‘Going from New Mexico to the Lower East Side was like going from heaven to hell.’ Photograph: Danny Lyon

In late 1965, fresh from photographing the nascent Civil Rights Movement in the American south, Danny Lyon joined a motorcycle gang called the Chicago Outlaws. He was 23 and already a "fully fledged romantic" who believed the best way to photograph a subject was from the inside out.

"I honestly didn't see any contradictions in what I was doing back then, but I sure as hell didn't sit down with bikers over a beer and discuss Martin Luther King and the struggle for black equality," he says now, laughing heartily. "These were guys who wore Iron Crosses. Hell, I even had one on my motorcycle. But they were outsiders and I was drawn to outsiders. From my involvement in the civil rights struggle, I knew the best way to get good pictures was to get involved. I was a participant who also happened to be a photographer."

Muhammad Ali in Florida, 1970 – a shoot he nearly turned down. Photograph: Danny Lyon

Lyon's insider approach to photojournalism has often been compared to the New Journalism of the late 1960s and early 1970s as espoused by the likes of Tom Wolfe, Truman Capote and Hunter S Thompson. "I think what I was doing preceded them by some years, but there were certainly some similarities," says Lyon. "I did once receive a letter from Hunter, in which he basically said I was crazy for joining the Chicago Outlaws."

The letter is included in Lyon's new book, The Seventh Dog, a retrospective which he describes as "a history of my life in photography". It was written when Thompson was researching what would become his first book, Hells Angels: The Strange and Terrible Saga of the Outlaw Motorcycle Gangs. It begins: "Dear Danny, I think you should get to hell out of that club unless it's absolutely necessary for photo action." Lyon chuckles as he repeats the line verbatim, adding: "He got that last bit right anyway. I did what was necessary for the photographs."

Now 72 and living "out in the desert in New Mexico between Santa Fe and Albuquerque", Lyon has spent 50 years flying in the face of the notion that reportage documentary photojournalism should be detached and objective. "From the start, I saw photography as a powerful political tool," he says, and the trajectory of his work – from those early images of the civil rights struggle in the early- to mid-60s to his insider documentation of the Occupy Wall Street protests of 2011 – testifies to this uncompromising stance.

Self-portrait (detail) Photograph: Danny Lyon

But there is a lot more to Lyon than that. As The Seventh Dog attests (the title is a reference to a recently acquired collie called Trip), he is a consummate book artist who merges personal snapshots and collages with single images from his documentary career in a style that is instantly recognisable from previous equally beautiful books such as Knave of Hearts and Memories of Myself. This time around, he has chosen to sequence the book in a reverse chronology, beginning with this recent work and steadily taking the reader on a journey back to the 1960s. "That way, the images just get better and better and more beautiful to look at – that's just the nature of photography."

In his attention to layout, design and sequencing he was ahead of the game, and as the importance of the photobook as an art object in itself has become central to the history and understanding of photography, his early first editions are now much sought after by collectors. "It's kind of ironic given that, for a long time, I was struggling to get stuff published," he says without bitterness. "The Bikeriders [the now iconic book that he made after a year with the Chicago Outlaws] was made to be sold cheaply – $2.95 for the softback, $6 for the hardback – so people would buy it, but it still ended up remaindered. [A first edition now changes hands for up to £1,500.] Same with my prison book, Conversations with the Dead, which has been out of print in America since 1972."

On the funeral route of the four girls murdered in Alabama, 1963. Photograph: Danny Lyon

Conversations with the Dead is the book that cemented Lyon's reputation and is still regarded by many critics and photographers alike as his most powerful statement, an often visceral exposé of the relentless cruelty of the American penal system. He spent a year photographing inside the Texan prison system with, he now says, "the intention of destroying that system". While doing so, he befriended several inmates including James Ray Renton, who had been convicted of the murder of a Texas police officer.

Lyon testified on Renton's behalf as a character witness at his murder trial in 1979 and, almost 30 years later, published Like a Thief's Dream, a memoir of their long friendship that also includes Renton's somewhat opaque account of what happened on the night of the killing. "To some, he's idealising people who really are not good people at all – they're just criminals," Lyon's friend, the novelist Larry McMurtry, said of him in an interview with the New York Times in 2009. "But to Danny maybe they're good people who just never had a chance. He hasn't really changed his principles any at all since he was young, when I first met him. He's an idealist, to a large extent."

John Lennon in 1970 Photograph: Danny Lyon

Lyon's idealism and non-conformist attitude have made him a kind of legend to several generations of younger photographers. Susan Meiselas once recalled meeting Lyon just after she had joined the famous Magnum photo-agency in 1976: "He greeted me by saying that he had just stolen my book, Carnival Strippers, from a local bookstore."

His rebelliousness may itself be a sustained reaction to the respectability of his middle-class upbringing. The son of German immigrants, he was raised in Kew Gardens in Queens, New York, where his father was a doctor. One of his father's patients was the great photographer Alfred Stieglitz, which Lyon interprets "as a sign, a kind of omen".

Lyon's first real exposure to the power of photography came later though, when as a teenager he bought a remaindered copy of Let Us Now Praise Famous Men, a classic of American photojournalism by Walker Evans and the campaigning journalist James Agee. "To tell the truth, Agee's writing had a more profound effect on me at the time than Evans's photographs," elaborates Lyon. "Agee had an unshakeable belief in documentary photography and film as a powerful instrument of truth. I'm old-fashioned; I still believe in his idea of pure realism." InMemories of Myself, which Lyon describes as "a perfect complement toThe Seventh Dog", he included an early series of portraits he shot in the late 1960s, when he made a kind of artistic pilgrimage to James Agee's hometown, Knoxville, Tennessee.

'Crossing the Ohio, Louisville', 1966. Hunter S Thompson urged Lyon not to join a biker gang. Photograph: Danny Lyon

Lyon's first camera was an Exa single-lens reflex bought on a family trip back to East Germany in the late 1950s . "I looked through it and soon discovered it was the only thing I was good at." At 22, he hitchhiked south and joined the Student Nonviolent Coordinating Committee (SNCC). A week later he was in jail in Albany, Georgia, in a cell next to the Rev Martin Luther King. "I was a self-taught photographer with an instinctive gift, but the SNCC made me a journalist," he says. "Suddenly I was at the centre of the biggest story of the decade. My life was interrupted by the Civil Rights struggle and I was seduced by it. Years later, at a talk I was giving, a black activist stood up and said: 'Our struggle put you on the map.' It irked me at the time, but in a way he was right."

His images from that turbulent time were published in his first book, The Movement, in 1963, but, by the following year, as more photographers headed south to cover the struggle, Lyon had moved on, returning east to Chicago, where he had attended college. There, he bought a Triumph TR6 motorcycle and joined the Chicago Outlaws, publishing The Bikeriders in 1968 – an unapologetically romantic but edgy book of photographs and edited conversations with the gang members. The book nodded to the macho rebellion of the 1950s cult film The Wild One, starring Marlon Brando, and prefigured the hippy biker film Easy Rider, released the following year, most notably in its most famous image, "Crossing the Ohio, Louisville" – a visual ode to the romance of dirty leather and polished chrome. "I recently found a letter I had written to a publisher back then and it sets out all my plans for the book. It's pretty close to what was actually created, but it's written by a juvenile who has trouble with his spelling and comes across as a kind of crazy person who writes in this sub-Jack Kerouac-style prose about the open road and the freedom of being an outlaw." Did the reality live up to the myth? "Hell, no. I was kind of horrified by the end. I remember I had a big disagreement with this guy who rolled out a huge Nazi flag as a picnic rug to put our beers on. By then I had realised that some of these guys were not so romantic after all. Then, when there was a race riot in Chicago, these big macho guys were scared to leave the club house in case they would be attacked by black people. I just thought it was laughable."

The Occupy protest in LA in 2011 Photograph: Danny Lyon

A book called The Destruction of Lower Manhattan followed, detailing the razing of 24 blocks of historic New York buildings in a development surge in 1967, which led, among other things, to the building of the World Trade Center. That book, he says, helped him gain access to the East Texan penitentiaries where he created Conversations with the Dead. "Back then, the prison population of Texas was around 12,000 – now, its over 200,000," he says, ruefully. "So, my book didn't work in the way I wanted it to."

At the time, his aim was not just to expose the US prison system, but, as he once put it, to "destroy Life magazine" and, by extension, all it stood for in terms of balanced journalism. He has mellowed somewhat since, but remains a rebel. He has moved with the times, making documentary films, including one on the Occupy movement, and has a fascinating blog, bleakbeauty.com, that suggests he is still an activist of sorts, albeit a reclusive one.

"I still take photographs, I make films and I sell alfalfa. I live in the desert and fish in the Rio Grande, which is half an hour from my house. Fishing allows you to think and reflect, but what I think about mostly is the river and how low it seems. I'm seeing the change in our climate first-hand and I'm living in a time where there is an inability in our democracy to react to it in any way. That makes me angry all over again."

In The Seventh Dog he reprints part of a lecture he gave at Stanford University in 2009 entitled "The End of the Age of Photography". Does he really think that we are living in the end times for photography, or at least the kind of socially engaged, committed documentary photography that he epitomised? "Well, the digital age has seen a certain kind of meaningless photography spread like a malignancy. There were always too many photographs, but now there's a kind of visual pollution. Digital technology has also been incredibly destructive. There's no getting around that. It has displaced the processes of photography, the film, the dark rooms, the paper, the camera stores and all the jobs those processes provided."

He falls silent for a moment. "I'm a photo-journalist who became an artist. I make prints by hand, I spend time on putting books together so they look beautiful. These things are important. They are evidence of the creative life you have lived. You can hold them in your hands. They're solid." And, in Danny Lyon's case, they carry the weight of a life lived defiantly out of step with convention.