Does the physical location of assembling to worship matter? Yes!
Does your group assemble to worship him in places dedicated to false religions? Then it is doing wrong!
A COMMAND AND WARNING
Exodus 34: 11-16 " Observe thou that which I command thee this day: behold, I drive out before thee the Amorite, and the Canaanite, and the Hittite, and the Perizzite, and the Hivite, and the Jebusite. Take heed to thyself, lest thou make a covenant with the inhabitants of the land whither thou goest, lest it be for a snare in the midst of thee: But ye shall destroy their altars, break their images, and cut down their groves: For thou shalt worship no other god: for the LORD, whose name is Jealous, is a jealous God: Lest thou make a covenant with the inhabitants of the land, and they go a whoring after their gods, and do sacrifice unto their gods, and one call thee, and thou eat of his sacrifice; And thou take of their daughters unto thy sons, and their daughters go a whoring after their gods, and make thy sons go a whoring after their gods."
Deuteronomy 7: 1-5 " When the LORD thy God shall bring thee into the land whither thou goest to possess it, and hath cast out many nations before thee, the Hittites, and the Girgashites, and the Amorites, and the Canaanites, and the Perizzites, and the Hivites, and the Jebusites, seven nations greater and mightier than thou; And when the LORD thy God shall deliver them before thee; thou shalt smite them, and utterly destroy them; thou shalt make no covenant with them, nor shew mercy unto them: Neither shalt thou make marriages with them; thy daughter thou shalt not give unto his son, nor his daughter shalt thou take unto thy son. For they will turn away thy son from following me, that they may serve other gods: so will the anger of the LORD be kindled against you, and destroy thee suddenly. But thus shall ye deal with them; ye shall destroy their altars, and break down their images, and cut down their groves, and burn their graven images with fire."
Deuteronomy 12: 1-3 " These are the statutes and judgments, which ye shall observe to do in the land, which the LORD God of thy fathers giveth thee to possess it, all the days that ye live upon the earth. Ye shall utterly destroy all the places, wherein the nations which ye shall possess served their gods, upon the high mountains, and upon the hills, and under every green tree: And ye shall overthrow their altars, and break their pillars, and burn their groves with fire; and ye shall hew down the graven images of their gods, and destroy the names of them out of that place."
The Lord's command was to destroy the places where false gods were worshiped and warned against making agreements with the people of the land, which includes making agreements not to rent or use their places. When renting these type of places, are you not helping them out financially to spread the false doctrines. God made it clear not to have anything to do with these pagan religions. God warned that these places of worship would be an influence, so they were to completely separate themselves from other religious practices.
THE TEMPLE OF GOD
1Corinthians 3: 17 " Know ye not that ye are the temple of God, and that the Spirit of God dwelleth in you? If any man defile the temple of God, him shall God destroy; for the temple of God is holy, which temple ye are."
Should we as the temples of God worship within the temples of the ungodly?
THE COUNTER
But we know the true God and are worshipping him alone.
1Corinthians 8: 9-11 " But take heed lest by any means this liberty of yours become a stumbling block to them that are weak. For if any man see thee which hast knowledge sit at meat in the idol's temple, shall not the conscience of him which is weak be emboldened to eat those things which are offered to idols; And through thy knowledge shall the weak brother perish, for whom Christ died?"
Yes, one should be strong enough in faith to understand and stand firm against paganism, but those that are weak, or new, who are facing trials may end up stumbling into the hands of the group whose building you are using. Is it worth the risk?
1Corinthians 10: 20-21 " But I say, that the things which the Gentiles sacrifice, they sacrifice to devils, and not to God: and I would not that ye should have fellowship with devils. Ye cannot drink the cup of the Lord, and the cup of devils: ye cannot be partakers of the Lord's table, and of the table of devils.2Corinthians 6:14-17 "Be ye not unequally yoked together with unbelievers: for what fellowship hath righteousness with unrighteousness? and what communion hath light with darkness? And what concord hath Christ with Belial? or what part hath he that believeth with an infidel? And what agreement hath the temple of God with idols? for ye are the temple of the living God; as God hath said, I will dwell in them, and walk in them; and I will be their God, and they shall be my people. Wherefore come out from among them, and be ye separate, saith the Lord, and touch not the unclean thing; and I will receive you."
SUMMARY
Scripture is full of examples showing that, when they refused to obey God's instructions they always were punished for their disobedience.