Rosen was born into a Jewish family in Harrow, Middlesex, the son of communist parents. His father Harold (1919–2008) was born in Brockton, Massachusetts. His mother was Connie (née Isakofsky) Rosen (1920-1976). Harold Rosen settled in the East End of London at the age of two, when his mother returned to the country of her birth. While a member of the Young Communist League, Harold met Connie Isakofsky, his future wife and Rosen's mother, in 1935. Harold was a secondary school teacher before becoming a professor of English at the Institute of Education in London, and Connie was a primary school teacher before becoming a training college lecturer; she also broadcast for the BBC. Producing a programme featuring poetry, she persuaded her son to write for it, and used some of the material he submitted. Their ancestors came from Poland, Russia, and Romania. Rosen was raised in Pinner, Middlesex, and went to various state schools in Pinner and Harrow, and Watford Grammar School for Boys;[3] having discovered the range of Jonathan Miller, he thought "wouldn't it be wonderful to know all about science, and know all about art, and be funny and urbane and all that". Subsequently, in his own words:

I went to Middlesex Hospital Medical School, started on the first part of a medical training, jacked it in and went on to do a degree in English at Oxford University. I then worked for the BBC until they chucked me out and I have been a freelance writer, broadcaster, lecturer, performer ever since – that's to say since 1972. Most of my books have been for children, but that's not how I started out. Sometime around the age of twelve and thirteen I began to get a sense that I liked writing, liked trying out different kinds of writing, I tried writing satirical poems about people I knew.