Mr Mattis's comments drew a pointed reaction from a Chinese official at the meeting.
Beijing's military buildup in the South China Sea calls into question its broader goals in the region, the U.S. secretary of defense told a high-profile global security forum on Saturday.
"I believe there are much larger consequences in the future when nations lose the rapport of their neighbours. eventually these (actions) do not pay off", Mattis said.
He said Beijing was intimidating and coercing others in the region by putting weapons systems on manmade islands.
The comments came on the heels of a string of events that highlight the tension between the world's two biggest economies over the South China Sea's disputed waters.
The U.S. also issued a stern statement criticizing what they deem as the militarization of the hotly disputed islands in the South China Sea.More news: Billionaire Koch Brothers Launch Million Dollar Campaign Against Trump's Popular Tariffs
In May, the United States had disinvited China from participating in annual maritime exercises in the Pacific in protest against Beijing's militarisation of the South China Sea.
Gen Mattis said Beijing had deployed military hardware, including anti-ship missiles, surface-to-air missiles and electronic jammers to locations across the South China Sea.
He Lei, the head of the Chinese delegation to the dialogue, said the islands were Chinese territory and it was "a sovereign and legal right for China to place our army and military weapons there".
The U.S., he said, "will continue to pursue a constructive, results-oriented relationship with China, cooperation when possible, and competing vigorously where we must".
Critics say the patrols have little impact on Beijing and mask the lack of a broader strategy to deal with the growing influence of China.
He said the USA remained committed to ensuring free and open transit in the region, a hub for shipping and natural resources beset by overlapping claims from China and several of its neighbors including Vietnam and the Philippines.More news: More Breast Cancer Patients Can Skip Chemo - Gene Test
A study by Asia Maritime Transparency Initiative (AMTI) found a year ago that China had built military length airstrips on the previously uninhabited Fiery Cross, Mischief and Subi reefs in the Spratlys.
Discussion about the number of U.S troops in South Korea is subject to bilateral discussions between Washington and Seoul that are separate from the North Korea negotiations, Mattis said.
The remarks reflect concern that smaller states, including open economies like Singapore, risk being caught in the wash of the USA trade actions.
On Sunday, two United States warships sailed close to the Paracel Islands in a long-planned operation created to challenge China's claims.
The Pentagon chief gave only a brief mention of the negotiations for the planned June 12 summit in Singapore between Trump and North Korean leader Kim Jong Un. Trump's recent tariffs on steel and aluminum imports have targeted allies in Europe, plus Japan, Canada and Mexico, while the U.S.is engaged in an escalating war of words with China, risking a full-blown trade war.
Reuters first reported that two US Navy warships sailed near South China Sea islands claimed by China on Sunday, prompting anger in Beijing, which claims most of the strategic waters where Chinese, Japanese and some Southeast Asian navies operate.More news: Tesla 'quite likely' to meet Model 3 goal; Musk remains chairman
China has not sent high-level officials to the Shangri-La conference.
- Mexico aims tariffs at Trump country, sees NAFTA complications
- Former Ark City PD chief living in Guatemala safe after volcanic eruption
- Melania Mystery Continues: First Lady Won't Accompany President To G7 Or Singapore
- How to watch French Open 2018 clash online
- Details of Fred's Manchester United deal revealed
- LeBron James' eye remains bloodshot in Game 2 of 2018 NBA Finals
- No women make Forbes' list of top paid sport stars
- Trump criticizes Canada's lumber sector after imposing steel, aluminum tariffs
- Neymar returns with brilliant solo goal against Croatia
- Novak Djokovic beaten by Marco Cecchinato in French Open quarter-finals