Japan also expressed concern over increased activity in the East China Sea.
China risks triggering unintended conflict with Asian rivals through its aggressive stance in maritime disputes, Japan said Tuesday in an annual security assessment.
China's sweeping claims over the South China Sea, where it has built a series of artificial islands capable of supporting military operations despite overlapping claims from other nations, have stoked international alarm.
The region's superpower "continues to act in an assertive manner" and its actions "include dangerous acts that could cause unintended consequences", Tokyo said in a defence white paper.
Beijing is under pressure to respect a UN-backed tribunal's finding last month that there is no legal basis for its ambitions over the resource-rich South China Sea where the Philippines, Vietnam, Malaysia and others also lay claims.
The white paper said China was "poised to fulfill its unilateral demands without compromise" including efforts "to turn these coercive changes to the status quo into a fait accompli".
And it again called on Beijing to abide by the ruling of the tribunal, which China has denounced as a fraud.
Japan also expressed concern over increased activity in the East China Sea, where the two countries have competing claims to a group of small uninhabited islets called the Senkaku in Japan and the Diaoyus in China.
"Recently, China has been intensifying activities near the Senkaku Islands, such as its military aircraft flying southward closer to the islands," it said.
In the year to March 2016, Japan's air force scrambled jets 571 times against Chinese planes flying near Japanese airspace, an increase of 107 from the previous year, it added.
China lodged a "solemn" protest with Japan over the defence white paper, state broadcaster CCTV said in Beijing.
China's official Xinhua news agency, in a report from Tokyo, blasted the paper, accusing Japan of "making irresponsible remarks on China's national defence and China's normal and legal maritime activities in the East and South China Seas".
Last month the two countries were at loggerheads over accusations Japanese warplanes locked their fire control radar onto Chinese aircraft.
Beijing sparked alarm after it unilaterally established an air defence identification zone in the East China Sea in 2013, demanding all aircraft submit flight plans when traversing the zone which covers islands disputed with Tokyo and also claimed by Taipei.
US Defence Secretary Ashton Carter said in February that China's military presence in the South China Sea was increasing the risk of "miscalculation or conflict". Washington regularly sends warships into the area to assert the right to freedom of navigation.
And tensions have also grown over Indonesia's Natuna Islands in the South China Sea, where Chinese and Indonesian boats have clashed.
President Joko Widodo visited the Natunas on a warship in June with his cabinet to send a message that Jakarta is serious about defending the remote archipelago.
Japan's white paper also highlighted concerns over North Korea's nuclear programme, saying it was possible it has "achieved the miniaturisation of nuclear weapons and has developed nuclear warheads".
Since carrying out a fourth nuclear test in January, North Korea has claimed it miniaturised a nuclear warhead to fit on a missile and successfully tested an engine designed for an inter-continental ballistic missile that could reach the US mainland.