He is the first U.S. secretary of state to ever visit Hiroshima, where around 140,000 were killed when the U.S. dropped its atomic bomb in 1945, BBC News wrote.
Mr. Kerry was joined by foreign ministers from the G7 group of nations who are holding talks in the city.
They laid wreaths at the memorial and observed a minute of silence.
The ministers also visited the Bomb Dome, over which the A-bomb exploded, and the nearby Hiroshima museum, which tells the personal stories of people who died.
Read alsoRussian-Japanese conflict around Pacific island arises againMr. Kerry wrote in the museum guestbook that it was "a stark, harsh, compelling reminder not only of our obligation to end the threat of nuclear weapons, but to rededicate all our effort to avoid war itself."
Honored to be 1st Sec State to visit Hiroshima Peace Memorial Museum & Park. Here is what I wrote in the guestbook. pic.twitter.com/suQSxMCLs1— John Kerry (@JohnKerry) April 11, 2016
Mr. Kerry previously said his time in Hiroshima would "revisit the past and honor those who perished" but stressed that his trip was "about the present and the future."
It also comes amid efforts to strengthen the relationship between the U.S. and Japan, particularly with growing concern about China's assertiveness in territorial disputes in Asia, affecting Japan and other U.S. allies.