"People involved in disturbing harmony in the country are also taking advantage of it", he said.
The Prime Minister also called on all the parties to support the war against corruption, which has been initiated by the government.
PM Modi said it would have been very good had a consensus been reached on the presidential candidate.
According to Kumar, Modi told the all-party meeting that while Indians believe that the cow is like our mother, that does not let people take the law into their own hands.
At the meeting, sources said, Naresh Agrawal of the Samajwadi Party suggested that the Opposition parties should have such interactions at least once a month to coordinate their positions.
CPI leader D Raja said the opposition "will seek answers from the government on what they are doing to check rising incidents of cow vigilantism and lynchings in the country that pose grave danger to the social harmony of the country and the idea of India".
He said the government was "fully committed to maintain peace in Jammu and Kashmir and to wipe out the anti-national elements from the state" and reiterated that the path laid down by former Prime Minister Atal Bihari Vajpayee was being pursued.
Seeking the support of the opposition parties in tackling "communal violence" in the name of cow protection, Modi also warned against giving a political or communal colour to the issue.
Parliament session to begin today: Opposition likely to raise China issue, J and K, Darjeeling unrest, GST. "There are other issues also which will come up", he added.
A total of 18 Bills have been listed for passage in the Lok Sabha and the Rajya Sabha in the upcoming Monsoon session of Parliament, a research body, PRS Legislative Research said on Saturday.
The Opposition simultaneously challenged the Centre on issues that underline a deteriorating internal security situation, including the recent attack on Amarnath pilgrims. "This is impossible, unless the government plans to pass all these bills and grants in the din minus any discussion", he said.