From the point of view of food safety, you're taking a risk if you leave it outside the refrigerator for more than two hours. Unfortunately, bacteria grow and multiply at temperatures between 40 degrees and 140 degrees, and they flourish at room temperature.
To avoid food borne illness, all foods of animal origin should be kept either hotter than 140 degrees or colder than 40 degrees. If you know you won't be returning home directly after shopping, bring along an insulated bag or box to keep cold foods cold until you can get them into the refrigerator.